Devices, methods, and graphical user interfaces for adaptively providing audio outputs

ABSTRACT

A wearable audio output device is in a respective physical environment and in communication with an electronic device. While one or more audio properties of the respective physical environment satisfy first audio criteria, the wearable audio output device provides audio output corresponding to the first audio criteria, the audio output including: audio corresponding to audio content from the electronic device at a first device-content audio level; and audio corresponding to ambient sound from the respective physical environment at a first ambient-sound audio level. The wearable audio output device detects a change in the one or more audio properties of the respective physical environment, and in response to detecting the change in the one or more audio properties of the respective physical environment, provides audio corresponding to ambient sound from the respective physical environment at a second ambient-sound audio level that is different from the first ambient-sound audio level.

RELATED APPLICATION

This application claims priority to U.S. Provisional Application Ser.No. 62/843,891, filed May 6, 2019, which is incorporated by referenceherein in its entirety.

TECHNICAL FIELD

This relates generally to electronic devices with audio output devicessuch as wearable audio output devices, including but not limited toelectronic devices that provide audio outputs adaptively based onchanges in context of the audio output devices.

BACKGROUND

Audio output devices, including wearable audio output devices such asheadphones and earphones, are widely used to provide audio outputs to auser. But conventional methods of providing audio outputs arecumbersome, inefficient, and limited. In some cases, conventionalmethods provide audio outputs in a manner that hinders a user's abilityto interact with his surrounding physical environment. In some cases,audio outputs are provided in a static manner irrespective of a user'sposition, orientation, or movement. In some cases, audio outputs areprovided in a static manner irrespective of changes in ambient audio,thereby interfering with a user's ability to understand when spoken to,particularly when wearing headphones or earphones. In addition,conventional methods take longer and require more user interaction thannecessary to adjust audio outputs for such context changes, therebywasting energy. This latter consideration is particularly important inbattery-operated devices.

SUMMARY

According, there is a need for electronic devices with improved methodsand interfaces for providing audio outputs adaptively based on changesin context of the audio output devices. Such methods and interfacesoptionally complement or replace conventional methods of providing audiooutputs. Such methods and interfaces reduce the number, extent, and/ornature of the inputs from a user and produce a more efficienthuman-machine interface. For battery-operated devices, such methods andinterfaces conserve power and increase the time between battery charges.

The above deficiencies and other problems associated with providingaudio outputs are reduced or eliminated by the disclosed devices. Insome embodiments, the device is a desktop computer. In some embodiments,the device is portable (e.g., a notebook computer, tablet computer, orhandheld device). In some embodiments, the device is a personalelectronic device (e.g., a wearable electronic device, such as a watch).In some embodiments, the device has (and/or is in communication with)one or more audio output devices (e.g., wearable audio output devices,such as in-ear earphones, earbuds, over-ear headphones, etc.). In someembodiments, the device has (and/or is in communication with) atouchpad. In some embodiments, the device has (and/or is incommunication with) a touch-sensitive display (also known as a “touchscreen” or “touch-screen display”). In some embodiments, the device hasa graphical user interface (GUI), one or more processors, memory and oneor more modules, programs or sets of instructions stored in the memoryfor performing multiple functions. In some embodiments, the userinteracts with the GUI primarily through stylus and/or finger contactsand gestures on the touch-sensitive surface. In some embodiments, thefunctions optionally include image editing, drawing, presenting, wordprocessing, spreadsheet making, game playing, telephoning, videoconferencing, e-mailing, instant messaging, workout support, digitalphotographing, digital videoing, web browsing, digital music/audioplaying, note taking, and/or digital video playing. Executableinstructions for performing these functions are, optionally, included ina non-transitory computer readable storage medium or other computerprogram product configured for execution by one or more processors.

In accordance with some embodiments, a method is performed at anelectronic device including one or more pose sensors for detecting apose of a user of the electronic device relative to a first physicalenvironment. The electronic device is in communication with one or moreaudio output devices. The method includes, while a first pose of theuser meets first presentation criteria: providing audio content at afirst simulated spatial location relative to the user. The methodincludes detecting a change in the pose of the user from the first poseto a second pose; and, in response to detecting the change in the poseof the user, and in accordance with a determination that the second poseof the user does not meet the first presentation criteria: providingaudio content at a second simulated spatial location relative to theuser that is different from the first simulated spatial location.

In accordance with some embodiments, a method is performed at one ormore wearable audio output devices that are in a respective physicalenvironment and that are in communication with an electronic device. Themethod includes, while one or more audio properties of the respectivephysical environment satisfy first audio criteria, providing audiooutput corresponding to the first audio criteria. The audio outputincludes: audio corresponding to audio content from the electronicdevice at a first device-content audio level; and audio corresponding toambient sound from the respective physical environment at a firstambient-sound audio level. The method includes detecting a change in theone or more audio properties of the respective physical environment;and, in response to detecting the change in the one or more audioproperties of the respective physical environment, providing audiocorresponding to ambient sound from the respective physical environmentat a second ambient-sound audio level that is different from the firstambient-sound audio level.

In accordance with some embodiments, an electronic device includes or isin communication with one or more audio output devices, optionally oneor more pose sensors (e.g., to detect pose of the electronic device, ofthe audio output devices, or of a user of the electronic device relativeto a first physical environment), optionally one or more audio inputdevices, optionally a display, optionally a touch-sensitive surface orother input device, one or more processors, and memory storing one ormore programs; the one or more programs are configured to be executed bythe one or more processors and the one or more programs includeinstructions for performing or causing performance of the operations ofany of the methods described herein. In accordance with someembodiments, a computer readable storage medium has stored thereininstructions that, when executed by an electronic device as describedherein, cause the device to perform or cause performance of theoperations of any of the methods described herein. In accordance withsome embodiments, a graphical user interface on an electronic device asdescribed herein includes one or more of the elements displayed in anyof the methods described herein, which are updated in response toinputs, as described in any of the methods described herein. Inaccordance with some embodiments, an electronic device as describedherein includes means for performing or causing performance of theoperations of any of the methods described herein. In accordance withsome embodiments, an information processing apparatus, for use in anelectronic device as described herein, includes means for performing orcausing performance of the operations of any of the methods describedherein.

Thus, electronic devices that include or are in communication with oneor more audio output devices, optionally one or more pose sensors (e.g.,to detect pose of the electronic device, of the audio output devices, orof a user of the electronic device relative to a first physicalenvironment), optionally one or more audio input devices, optionally adisplay, and optionally a touch-sensitive surface or other input device,are provided with improved methods and interfaces for adaptivelyproviding audio outputs, thereby increasing the effectiveness,efficiency, and user satisfaction with such devices. Such methods andinterfaces may complement or replace conventional methods for providingaudio outputs.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various described embodiments,reference should be made to the Description of Embodiments below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures.

FIG. 1A is a block diagram illustrating a portable multifunction devicewith a touch-sensitive display in accordance with some embodiments.

FIG. 1B is a block diagram illustrating example components for eventhandling in accordance with some embodiments.

FIG. 2 illustrates a portable multifunction device having a touch screenin accordance with some embodiments.

FIG. 3A is a block diagram of an example multifunction device with adisplay and a touch-sensitive surface in accordance with someembodiments.

FIG. 3B is a block diagram of an example wearable audio output device inaccordance with some embodiments.

FIG. 3C illustrates example audio control by a wearable audio outputdevice in accordance with some embodiments.

FIG. 4A illustrates an example user interface for a menu of applicationson a portable multifunction device in accordance with some embodiments.

FIG. 4B illustrates an example user interface for a multifunction devicewith a touch-sensitive surface that is separate from the display inaccordance with some embodiments.

FIGS. 5A-5L illustrate example adaptive audio outputs in response toexample user interactions with wearable audio output devices inaccordance with some embodiments.

FIGS. 6A-6J illustrate example adaptive audio outputs in response toexample changes in the audio properties of a surrounding physicalenvironment in accordance with some embodiments.

FIGS. 7A-7B are flow diagrams of a process for adaptively changingsimulated spatial locations of audio outputs in response to changes inuser pose in accordance with some embodiments.

FIGS. 8A-8B are flow diagrams of a process for adaptively changing audiooutput levels in response to changes in the audio properties of asurrounding physical environment in accordance with some embodiments.

DESCRIPTION OF EMBODIMENTS

As noted above, audio output devices such as wearable audio outputdevices are widely used to provide audio outputs to a user. Many audiooutput devices provide audio outputs in a static manner that does notadapt to changes in a user's pose or changes in the audio properties ofthe surrounding physical environment. The methods, devices, and userinterfaces/interactions described herein improve how audio outputs areprovided in multiple ways. For example, embodiments disclosed hereindescribe ways to provide audio outputs adaptively based on changes incontext of the audio output devices, such as changes in the user's poseor changes in the audio properties of the surrounding physicalenvironment, so that the user can better interact with his surroundingphysical environment.

Below, FIGS. 1A-1B, 2, and 3A-3C provide a description of exampledevices and examples of their operation. FIGS. 4A-4B illustrate exampleuser interface for example devices on which the embodiments disclosedherein are implemented. FIGS. 5A-5L illustrate example adaptive audiooutputs in response to example user interactions with wearable audiooutput devices. FIGS. 6A-6J illustrate example adaptive audio outputs inresponse to example changes in the audio properties of a surroundingphysical environment. FIGS. 7A-7B illustrate a flow diagram of a methodof adaptively changing simulated spatial locations of audio outputs inresponse to changes in user pose. FIGS. 8A-8B illustrate a flow diagramof a method of adaptively changing audio output levels in response tochanges in the audio properties of a surrounding physical environment.The user interfaces in FIGS. 5A-5L and 6A-6J are used to illustrate theprocesses in FIGS. 7A-7B and 8A-8B.

EXAMPLE DEVICES

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings. In the following detaileddescription, numerous specific details are set forth in order to providea thorough understanding of the various described embodiments. However,it will be apparent to one of ordinary skill in the art that the variousdescribed embodiments may be practiced without these specific details.In other instances, well-known methods, procedures, components,circuits, and networks have not been described in detail so as not tounnecessarily obscure aspects of the embodiments.

It will also be understood that, although the terms first, second, etc.are, in some instances, used herein to describe various elements, theseelements should not be limited by these terms. These terms are only usedto distinguish one element from another. For example, a first audiooutput could be termed a second audio output, and, similarly, a secondaudio output could be termed a first audio output, without departingfrom the scope of the various described embodiments. The first audiooutput and the second audio output are both audio outputs, but they arenot the same audio output, unless the context clearly indicatesotherwise.

The terminology used in the description of the various describedembodiments herein is for the purpose of describing particularembodiments only and is not intended to be limiting. As used in thedescription of the various described embodiments and the appendedclaims, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

As used herein, the term “if” is, optionally, construed to mean “when”or “upon” or “in response to determining” or “in response to detecting,”depending on the context. Similarly, the phrase “if it is determined” or“if [a stated condition or event] is detected” is, optionally, construedto mean “upon determining” or “in response to determining” or “upondetecting [the stated condition or event]” or “in response to detecting[the stated condition or event],” depending on the context.

Embodiments of electronic devices, user interfaces for such devices, andassociated processes for using such devices are described. In someembodiments, the device is a portable communications device, such as amobile telephone, that also contains other functions, such as PDA and/ormusic player functions. Example embodiments of portable multifunctiondevices include, without limitation, the iPhone®, iPod Touch®, and iPad®devices from Apple Inc. of Cupertino, Calif. Other portable electronicdevices, such as laptops or tablet computers with touch-sensitivesurfaces (e.g., touch-screen displays and/or touchpads), are,optionally, used. It should also be understood that, in someembodiments, the device is not a portable communications device, but isa desktop computer with a touch-sensitive surface (e.g., a touch-screendisplay and/or a touchpad).

In the discussion that follows, an electronic device that includes adisplay and a touch-sensitive surface is described. It should beunderstood, however, that the electronic device optionally includes oneor more other physical user-interface devices, such as a physicalkeyboard, a mouse and/or a joystick.

The device typically supports a variety of applications, such as one ormore of the following: a note taking application, a drawing application,a presentation application, a word processing application, a websitecreation application, a disk authoring application, a spreadsheetapplication, a gaming application, a telephone application, a videoconferencing application, an e-mail application, an instant messagingapplication, a workout support application, a photo managementapplication, a digital camera application, a digital video cameraapplication, a web browsing application, a digital music playerapplication, and/or a digital video player application.

The various applications that are executed on the device optionally useat least one common physical user-interface device, such as thetouch-sensitive surface. One or more functions of the touch-sensitivesurface as well as corresponding information displayed on the deviceare, optionally, adjusted and/or varied from one application to the nextand/or within a respective application. In this way, a common physicalarchitecture (such as the touch-sensitive surface) of the deviceoptionally supports the variety of applications with user interfacesthat are intuitive and transparent to the user.

Attention is now directed toward embodiments of portable devices withtouch-sensitive displays. FIG. 1A is a block diagram illustratingportable multifunction device 100 with touch-sensitive display system112 in accordance with some embodiments. Touch-sensitive display system112 is sometimes called a “touch screen” for convenience, and issometimes simply called a touch-sensitive display. Device 100 includesmemory 102 (which optionally includes one or more computer readablestorage mediums), memory controller 122, one or more processing units(CPUs) 120, peripherals interface 118, RF circuitry 108, audio circuitry110, speaker 111, microphone 113, input/output (I/O) subsystem 106,other input or control devices 116, and external port 124. Device 100optionally includes one or more optical sensors 164. Device 100optionally includes one or more intensity sensors 165 for detectingintensities of contacts on device 100 (e.g., a touch-sensitive surfacesuch as touch-sensitive display system 112 of device 100). Device 100optionally includes one or more tactile output generators 167 forgenerating tactile outputs on device 100 (e.g., generating tactileoutputs on a touch-sensitive surface such as touch-sensitive displaysystem 112 of device 100 or touchpad 355 of device 300). Thesecomponents optionally communicate over one or more communication busesor signal lines 103.

As used in the specification and claims, the term “tactile output”refers to physical displacement of a device relative to a previousposition of the device, physical displacement of a component (e.g., atouch-sensitive surface) of a device relative to another component(e.g., housing) of the device, or displacement of the component relativeto a center of mass of the device that will be detected by a user withthe user's sense of touch. For example, in situations where the deviceor the component of the device is in contact with a surface of a userthat is sensitive to touch (e.g., a finger, palm, or other part of auser's hand), the tactile output generated by the physical displacementwill be interpreted by the user as a tactile sensation corresponding toa perceived change in physical characteristics of the device or thecomponent of the device. For example, movement of a touch-sensitivesurface (e.g., a touch-sensitive display or trackpad) is, optionally,interpreted by the user as a “down click” or “up click” of a physicalactuator button. In some cases, a user will feel a tactile sensationsuch as an “down click” or “up click” even when there is no movement ofa physical actuator button associated with the touch-sensitive surfacethat is physically pressed (e.g., displaced) by the user's movements. Asanother example, movement of the touch-sensitive surface is, optionally,interpreted or sensed by the user as “roughness” of the touch-sensitivesurface, even when there is no change in smoothness of thetouch-sensitive surface. While such interpretations of touch by a userwill be subject to the individualized sensory perceptions of the user,there are many sensory perceptions of touch that are common to a largemajority of users. Thus, when a tactile output is described ascorresponding to a particular sensory perception of a user (e.g., an “upclick,” a “down click,” “roughness”), unless otherwise stated, thegenerated tactile output corresponds to physical displacement of thedevice or a component thereof that will generate the described sensoryperception for a typical (or average) user. Using tactile outputs toprovide haptic feedback to a user enhances the operability of the deviceand makes the user-device interface more efficient (e.g., by helping theuser to provide proper inputs and reducing user mistakes whenoperating/interacting with the device) which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In some embodiments, a tactile output pattern specifies characteristicsof a tactile output, such as the amplitude of the tactile output, theshape of a movement waveform of the tactile output, the frequency of thetactile output, and/or the duration of the tactile output.

When tactile outputs with different tactile output patterns aregenerated by a device (e.g., via one or more tactile output generatorsthat move a moveable mass to generate tactile outputs), the tactileoutputs may invoke different haptic sensations in a user holding ortouching the device. While the sensation of the user is based on theuser's perception of the tactile output, most users will be able toidentify changes in waveform, frequency, and amplitude of tactileoutputs generated by the device. Thus, the waveform, frequency andamplitude can be adjusted to indicate to the user that differentoperations have been performed. As such, tactile outputs with tactileoutput patterns that are designed, selected, and/or engineered tosimulate characteristics (e.g., size, material, weight, stiffness,smoothness, etc.); behaviors (e.g., oscillation, displacement,acceleration, rotation, expansion, etc.); and/or interactions (e.g.,collision, adhesion, repulsion, attraction, friction, etc.) of objectsin a given environment (e.g., a user interface that includes graphicalfeatures and objects, a simulated physical environment with virtualboundaries and virtual objects, a real physical environment withphysical boundaries and physical objects, and/or a combination of any ofthe above) will, in some circumstances, provide helpful feedback tousers that reduces input errors and increases the efficiency of theuser's operation of the device. Additionally, tactile outputs are,optionally, generated to correspond to feedback that is unrelated to asimulated physical characteristic, such as an input threshold or aselection of an object. Such tactile outputs will, in somecircumstances, provide helpful feedback to users that reduces inputerrors and increases the efficiency of the user's operation of thedevice.

In some embodiments, a tactile output with a suitable tactile outputpattern serves as a cue for the occurrence of an event of interest in auser interface or behind the scenes in a device. Examples of the eventsof interest include activation of an affordance (e.g., a real or virtualbutton, or toggle switch) provided on the device or in a user interface,success or failure of a requested operation, reaching or crossing aboundary in a user interface, entry into a new state, switching of inputfocus between objects, activation of a new mode, reaching or crossing aninput threshold, detection or recognition of a type of input or gesture,etc. In some embodiments, tactile outputs are provided to serve as awarning or an alert for an impending event or outcome that would occurunless a redirection or interruption input is timely detected. Tactileoutputs are also used in other contexts to enrich the user experience,improve the accessibility of the device to users with visual or motordifficulties or other accessibility needs, and/or improve efficiency andfunctionality of the user interface and/or the device. Tactile outputsare optionally accompanied with audio outputs and/or visible userinterface changes, which further enhance a user's experience when theuser interacts with a user interface and/or the device, and facilitatebetter conveyance of information regarding the state of the userinterface and/or the device, and which reduce input errors and increasethe efficiency of the user's operation of the device.

It should be appreciated that device 100 is only one example of aportable multifunction device, and that device 100 optionally has moreor fewer components than shown, optionally combines two or morecomponents, or optionally has a different configuration or arrangementof the components. The various components shown in FIG. 1A areimplemented in hardware, software, firmware, or a combination thereof,including one or more signal processing and/or application specificintegrated circuits.

Memory 102 optionally includes high-speed random access memory andoptionally also includes non-volatile memory, such as one or moremagnetic disk storage devices, flash memory devices, or othernon-volatile solid-state memory devices. Access to memory 102 by othercomponents of device 100, such as CPU(s) 120 and the peripheralsinterface 118, is, optionally, controlled by memory controller 122.

Peripherals interface 118 can be used to couple input and outputperipherals of the device to CPU(s) 120 and memory 102. The one or moreprocessors 120 run or execute various software programs and/or sets ofinstructions stored in memory 102 to perform various functions fordevice 100 and to process data.

In some embodiments, peripherals interface 118, CPU(s) 120, and memorycontroller 122 are, optionally, implemented on a single chip, such aschip 104. In some other embodiments, they are, optionally, implementedon separate chips.

RF (radio frequency) circuitry 108 receives and sends RF signals, alsocalled electromagnetic signals. RF circuitry 108 converts electricalsignals to/from electromagnetic signals and communicates withcommunications networks and other communications devices via theelectromagnetic signals. RF circuitry 108 optionally includes well-knowncircuitry for performing these functions, including but not limited toan antenna system, an RF transceiver, one or more amplifiers, a tuner,one or more oscillators, a digital signal processor, a CODEC chipset, asubscriber identity module (SIM) card, memory, and so forth. RFcircuitry 108 optionally communicates with networks, such as theInternet, also referred to as the World Wide Web (WWW), an intranetand/or a wireless network, such as a cellular telephone network, awireless local area network (LAN) and/or a metropolitan area network(MAN), and other devices by wireless communication. The wirelesscommunication optionally uses any of a plurality of communicationsstandards, protocols and technologies, including but not limited toGlobal System for Mobile Communications (GSM), Enhanced Data GSMEnvironment (EDGE), high-speed downlink packet access (HSDPA),high-speed uplink packet access (HSDPA), Evolution, Data-Only (EV-DO),HSPA, HSPA+, Dual-Cell HSPA (DC-HSPA), long term evolution (LTE), nearfield communication (NFC), wideband code division multiple access(W-CDMA), code division multiple access (CDMA), time division multipleaccess (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a,IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g and/or IEEE802.11n), voice over Internet Protocol (VoIP), Wi-MAX, a protocol fore-mail (e.g., Internet message access protocol (IMAP) and/or post officeprotocol (POP)), instant messaging (e.g., extensible messaging andpresence protocol (XMPP), Session Initiation Protocol for InstantMessaging and Presence Leveraging Extensions (SIMPLE), Instant Messagingand Presence Service (IMPS)), and/or Short Message Service (SMS), or anyother suitable communication protocol, including communication protocolsnot yet developed as of the filing date of this document.

Audio circuitry 110, speaker 111, and microphone 113 provide an audiointerface between a user and device 100. Audio circuitry 110 receivesaudio data from peripherals interface 118, converts the audio data to anelectrical signal, and transmits the electrical signal to speaker 111.Speaker 111 converts the electrical signal to human-audible sound waves.Audio circuitry 110 also receives electrical signals converted bymicrophone 113 from sound waves. Audio circuitry 110 converts theelectrical signal to audio data and transmits the audio data toperipherals interface 118 for processing. Audio data is, optionally,retrieved from and/or transmitted to memory 102 and/or RF circuitry 108by peripherals interface 118. In some embodiments, audio circuitry 110also includes a headset jack (e.g., 212, FIG. 2). The headset jackprovides an interface between audio circuitry 110 and removable audioinput/output peripherals, such as output-only headphones or a headsetwith both output (e.g., a headphone for one or both ears) and input(e.g., a microphone).

I/O subsystem 106 couples input/output peripherals on device 100, suchas touch-sensitive display system 112 and other input or control devices116, with peripherals interface 118. I/O subsystem 106 optionallyincludes display controller 156, optical sensor controller 158,intensity sensor controller 159, haptic feedback controller 161, and oneor more input controllers 160 for other input or control devices. Theone or more input controllers 160 receive/send electrical signalsfrom/to other input or control devices 116. The other input or controldevices 116 optionally include physical buttons (e.g., push buttons,rocker buttons, etc.), dials, slider switches, joysticks, click wheels,and so forth. In some alternate embodiments, input controller(s) 160are, optionally, coupled with any (or none) of the following: akeyboard, infrared port, USB port, stylus, and/or a pointer device suchas a mouse. The one or more buttons (e.g., 208, FIG. 2) optionallyinclude an up/down button for volume control of speaker 111 and/ormicrophone 113. The one or more buttons optionally include a push button(e.g., 206, FIG. 2).

Touch-sensitive display system 112 provides an input interface and anoutput interface between the device and a user. Display controller 156receives and/or sends electrical signals from/to touch-sensitive displaysystem 112. In some embodiments, touch-sensitive display system 112 ordisplay controller 156, or a combination of touch-sensitive display 112and display controller 156, are referred to as a display generationcomponent of device 100. Touch-sensitive display system 112 displaysvisual output to the user. The visual output optionally includesgraphics, text, icons, video, and any combination thereof (collectivelytermed “graphics”). In some embodiments, some or all of the visualoutput corresponds to user interface objects. As used herein, the term“affordance” refers to a user-interactive graphical user interfaceobject (e.g., a graphical user interface object that is configured torespond to inputs directed toward the graphical user interface object).Examples of user-interactive graphical user interface objects include,without limitation, a button, slider, icon, selectable menu item,switch, hyperlink, or other user interface control.

Touch-sensitive display system 112 has a touch-sensitive surface, sensoror set of sensors that accepts input from the user based on hapticand/or tactile contact. Touch-sensitive display system 112 and displaycontroller 156 (along with any associated modules and/or sets ofinstructions in memory 102) detect contact (and any movement or breakingof the contact) on touch-sensitive display system 112 and converts thedetected contact into interaction with user-interface objects (e.g., oneor more soft keys, icons, web pages or images) that are displayed ontouch-sensitive display system 112. In some embodiments, a point ofcontact between touch-sensitive display system 112 and the usercorresponds to a finger of the user or a stylus.

Touch-sensitive display system 112 optionally uses LCD (liquid crystaldisplay) technology, LPD (light emitting polymer display) technology, orLED (light emitting diode) technology, although other displaytechnologies are used in other embodiments. Touch-sensitive displaysystem 112 and display controller 156 optionally detect contact and anymovement or breaking thereof using any of a plurality of touch sensingtechnologies now known or later developed, including but not limited tocapacitive, resistive, infrared, and surface acoustic wave technologies,as well as other proximity sensor arrays or other elements fordetermining one or more points of contact with touch-sensitive displaysystem 112. In some embodiments, projected mutual capacitance sensingtechnology is used, such as that found in the iPhone®, iPod Touch®, andiPad® from Apple Inc. of Cupertino, Calif.

Touch-sensitive display system 112 optionally has a video resolution inexcess of 100 dpi. In some embodiments, the touch screen videoresolution is in excess of 400 dpi (e.g., 500 dpi, 800 dpi, or greater).The user optionally makes contact with touch-sensitive display system112 using any suitable object or appendage, such as a stylus, a finger,and so forth. In some embodiments, the user interface is designed towork with finger-based contacts and gestures, which can be less precisethan stylus-based input due to the larger area of contact of a finger onthe touch screen. In some embodiments, the device translates the roughfinger-based input into a precise pointer/cursor position or command forperforming the actions desired by the user.

In some embodiments, in addition to the touch screen, device 100optionally includes a touchpad (not shown) for activating ordeactivating particular functions. In some embodiments, the touchpad isa touch-sensitive area of the device that, unlike the touch screen, doesnot display visual output. The touchpad is, optionally, atouch-sensitive surface that is separate from touch-sensitive displaysystem 112 or an extension of the touch-sensitive surface formed by thetouch screen.

Device 100 also includes power system 162 for powering the variouscomponents. Power system 162 optionally includes a power managementsystem, one or more power sources (e.g., battery, alternating current(AC)), a recharging system, a power failure detection circuit, a powerconverter or inverter, a power status indicator (e.g., a light-emittingdiode (LED)) and any other components associated with the generation,management and distribution of power in portable devices.

Device 100 optionally also includes one or more optical sensors 164.FIG. 1A shows an optical sensor coupled with optical sensor controller158 in I/O subsystem 106. Optical sensor(s) 164 optionally includecharge-coupled device (CCD) or complementary metal-oxide semiconductor(CMOS) phototransistors. Optical sensor(s) 164 receive light from theenvironment, projected through one or more lens, and converts the lightto data representing an image. In conjunction with imaging module 143(also called a camera module), optical sensor(s) 164 optionally capturestill images and/or video. In some embodiments, an optical sensor islocated on the back of device 100, opposite touch-sensitive displaysystem 112 on the front of the device, so that the touch screen isenabled for use as a viewfinder for still and/or video imageacquisition. In some embodiments, another optical sensor is located onthe front of the device so that the user's image is obtained (e.g., forselfies, for videoconferencing while the user views the other videoconference participants on the touch screen, etc.).

Device 100 optionally also includes one or more contact intensitysensors 165. FIG. 1A shows a contact intensity sensor coupled withintensity sensor controller 159 in I/O subsystem 106. Contact intensitysensor(s) 165 optionally include one or more piezoresistive straingauges, capacitive force sensors, electric force sensors, piezoelectricforce sensors, optical force sensors, capacitive touch-sensitivesurfaces, or other intensity sensors (e.g., sensors used to measure theforce (or pressure) of a contact on a touch-sensitive surface). Contactintensity sensor(s) 165 receive contact intensity information (e.g.,pressure information or a proxy for pressure information) from theenvironment. In some embodiments, at least one contact intensity sensoris collocated with, or proximate to, a touch-sensitive surface (e.g.,touch-sensitive display system 112). In some embodiments, at least onecontact intensity sensor is located on the back of device 100, oppositetouch-screen display system 112 which is located on the front of device100.

Device 100 optionally also includes one or more proximity sensors 166.FIG. 1A shows proximity sensor 166 coupled with peripherals interface118. Alternately, proximity sensor 166 is coupled with input controller160 in I/O subsystem 106. In some embodiments, the proximity sensorturns off and disables touch-sensitive display system 112 when themultifunction device is placed near the user's ear (e.g., when the useris making a phone call).

Device 100 optionally also includes one or more tactile outputgenerators 167. FIG. 1A shows a tactile output generator coupled withhaptic feedback controller 161 in I/O subsystem 106. In someembodiments, tactile output generator(s) 167 include one or moreelectroacoustic devices such as speakers or other audio componentsand/or electromechanical devices that convert energy into linear motionsuch as a motor, solenoid, electroactive polymer, piezoelectricactuator, electrostatic actuator, or other tactile output generatingcomponent (e.g., a component that converts electrical signals intotactile outputs on the device). Tactile output generator(s) 167 receivetactile feedback generation instructions from haptic feedback module 133and generates tactile outputs on device 100 that are capable of beingsensed by a user of device 100. In some embodiments, at least onetactile output generator is collocated with, or proximate to, atouch-sensitive surface (e.g., touch-sensitive display system 112) and,optionally, generates a tactile output by moving the touch-sensitivesurface vertically (e.g., in/out of a surface of device 100) orlaterally (e.g., back and forth in the same plane as a surface of device100). In some embodiments, at least one tactile output generator sensoris located on the back of device 100, opposite touch-sensitive displaysystem 112, which is located on the front of device 100.

Device 100 optionally also includes one or more accelerometers 168. FIG.1A shows accelerometer 168 coupled with peripherals interface 118.Alternately, accelerometer 168 is, optionally, coupled with an inputcontroller 160 in I/O subsystem 106. In some embodiments, information isdisplayed on the touch-screen display in a portrait view or a landscapeview based on an analysis of data received from the one or moreaccelerometers. Device 100 optionally includes, in addition toaccelerometer(s) 168, a magnetometer (not shown) and a GPS (or GLONASSor other global navigation system) receiver (not shown) for obtaininginformation concerning the location and orientation (e.g., portrait orlandscape) of device 100.

In some embodiments, the software components stored in memory 102include operating system 126, communication module (or set ofinstructions) 128, contact/motion module (or set of instructions) 130,graphics module (or set of instructions) 132, haptic feedback module (orset of instructions) 133, text input module (or set of instructions)134, Global Positioning System (GPS) module (or set of instructions)135, and applications (or sets of instructions) 136. Furthermore, insome embodiments, memory 102 stores device/global internal state 157, asshown in FIGS. 1A and 3. Device/global internal state 157 includes oneor more of: active application state, indicating which applications, ifany, are currently active; display state, indicating what applications,views or other information occupy various regions of touch-sensitivedisplay system 112; sensor state, including information obtained fromthe device's various sensors and other input or control devices 116; andlocation and/or positional information concerning the device's locationand/or attitude.

Operating system 126 (e.g., iOS, Darwin, RTXC, LINUX, UNIX, OS X,WINDOWS, or an embedded operating system such as VxWorks) includesvarious software components and/or drivers for controlling and managinggeneral system tasks (e.g., memory management, storage device control,power management, etc.) and facilitates communication between varioushardware and software components.

Communication module 128 facilitates communication with other devicesover one or more external ports 124 and also includes various softwarecomponents for handling data received by RF circuitry 108 and/orexternal port 124. External port 124 (e.g., Universal Serial Bus (USB),FIREWIRE, etc.) is adapted for coupling directly to other devices orindirectly over a network (e.g., the Internet, wireless LAN, etc.). Insome embodiments, the external port is a multi-pin (e.g., 30-pin)connector that is the same as, or similar to and/or compatible with the30-pin connector used in some iPhone®, iPod Touch®, and iPad® devicesfrom Apple Inc. of Cupertino, Calif. In some embodiments, the externalport is a Lightning connector that is the same as, or similar to and/orcompatible with the Lightning connector used in some iPhone®, iPodTouch®, and iPad® devices from Apple Inc. of Cupertino, Calif.

Contact/motion module 130 optionally detects contact withtouch-sensitive display system 112 (in conjunction with displaycontroller 156) and other touch-sensitive devices (e.g., a touchpad orphysical click wheel). Contact/motion module 130 includes varioussoftware components for performing various operations related todetection of contact (e.g., by a finger or by a stylus), such asdetermining if contact has occurred (e.g., detecting a finger-downevent), determining an intensity of the contact (e.g., the force orpressure of the contact or a substitute for the force or pressure of thecontact), determining if there is movement of the contact and trackingthe movement across the touch-sensitive surface (e.g., detecting one ormore finger-dragging events), and determining if the contact has ceased(e.g., detecting a finger-up event or a break in contact).Contact/motion module 130 receives contact data from the touch-sensitivesurface. Determining movement of the point of contact, which isrepresented by a series of contact data, optionally includes determiningspeed (magnitude), velocity (magnitude and direction), and/or anacceleration (a change in magnitude and/or direction) of the point ofcontact. These operations are, optionally, applied to single contacts(e.g., one finger contacts or stylus contacts) or to multiplesimultaneous contacts (e.g., “multitouch”/multiple finger contacts). Insome embodiments, contact/motion module 130 and display controller 156detect contact on a touchpad.

Contact/motion module 130 optionally detects a gesture input by a user.Different gestures on the touch-sensitive surface have different contactpatterns (e.g., different motions, timings, and/or intensities ofdetected contacts). Thus, a gesture is, optionally, detected bydetecting a particular contact pattern. For example, detecting a fingertap gesture includes detecting a finger-down event followed by detectinga finger-up (lift off) event at the same position (or substantially thesame position) as the finger-down event (e.g., at the position of anicon). As another example, detecting a finger swipe gesture on thetouch-sensitive surface includes detecting a finger-down event followedby detecting one or more finger-dragging events, and subsequentlyfollowed by detecting a finger-up (lift off) event. Similarly, tap,swipe, drag, and other gestures are optionally detected for a stylus bydetecting a particular contact pattern for the stylus.

In some embodiments, detecting a finger tap gesture depends on thelength of time between detecting the finger-down event and the finger-upevent, but is independent of the intensity of the finger contact betweendetecting the finger-down event and the finger-up event. In someembodiments, a tap gesture is detected in accordance with adetermination that the length of time between the finger-down event andthe finger-up event is less than a predetermined value (e.g., less than0.1, 0.2, 0.3, 0.4 or 0.5 seconds), independent of whether the intensityof the finger contact during the tap meets a given intensity threshold(greater than a nominal contact-detection intensity threshold), such asa light press or deep press intensity threshold. Thus, a finger tapgesture can satisfy particular input criteria that do not require thatthe characteristic intensity of a contact satisfy a given intensitythreshold in order for the particular input criteria to be met. Forclarity, the finger contact in a tap gesture typically needs to satisfya nominal contact-detection intensity threshold, below which the contactis not detected, in order for the finger-down event to be detected. Asimilar analysis applies to detecting a tap gesture by a stylus or othercontact. In cases where the device is capable of detecting a finger orstylus contact hovering over a touch sensitive surface, the nominalcontact-detection intensity threshold optionally does not correspond tophysical contact between the finger or stylus and the touch sensitivesurface.

The same concepts apply in an analogous manner to other types ofgestures. For example, a swipe gesture, a pinch gesture, a depinchgesture, and/or a long press gesture are optionally detected based onthe satisfaction of criteria that are either independent of intensitiesof contacts included in the gesture, or do not require that contact(s)that perform the gesture reach intensity thresholds in order to berecognized. For example, a swipe gesture is detected based on an amountof movement of one or more contacts; a pinch gesture is detected basedon movement of two or more contacts towards each other; a depinchgesture is detected based on movement of two or more contacts away fromeach other; and a long press gesture is detected based on a duration ofthe contact on the touch-sensitive surface with less than a thresholdamount of movement. As such, the statement that particular gesturerecognition criteria do not require that the intensity of the contact(s)meet a respective intensity threshold in order for the particulargesture recognition criteria to be met means that the particular gesturerecognition criteria are capable of being satisfied if the contact(s) inthe gesture do not reach the respective intensity threshold, and arealso capable of being satisfied in circumstances where one or more ofthe contacts in the gesture do reach or exceed the respective intensitythreshold. In some embodiments, a tap gesture is detected based on adetermination that the finger-down and finger-up event are detectedwithin a predefined time period, without regard to whether the contactis above or below the respective intensity threshold during thepredefined time period, and a swipe gesture is detected based on adetermination that the contact movement is greater than a predefinedmagnitude, even if the contact is above the respective intensitythreshold at the end of the contact movement. Even in implementationswhere detection of a gesture is influenced by the intensity of contactsperforming the gesture (e.g., the device detects a long press morequickly when the intensity of the contact is above an intensitythreshold or delays detection of a tap input when the intensity of thecontact is higher), the detection of those gestures does not requirethat the contacts reach a particular intensity threshold so long as thecriteria for recognizing the gesture can be met in circumstances wherethe contact does not reach the particular intensity threshold (e.g.,even if the amount of time that it takes to recognize the gesturechanges).

Contact intensity thresholds, duration thresholds, and movementthresholds are, in some circumstances, combined in a variety ofdifferent combinations in order to create heuristics for distinguishingtwo or more different gestures directed to the same input element orregion so that multiple different interactions with the same inputelement are enabled to provide a richer set of user interactions andresponses. The statement that a particular set of gesture recognitioncriteria do not require that the intensity of the contact(s) meet arespective intensity threshold in order for the particular gesturerecognition criteria to be met does not preclude the concurrentevaluation of other intensity-dependent gesture recognition criteria toidentify other gestures that do have a criteria that is met when agesture includes a contact with an intensity above the respectiveintensity threshold. For example, in some circumstances, first gesturerecognition criteria for a first gesture—which do not require that theintensity of the contact(s) meet a respective intensity threshold inorder for the first gesture recognition criteria to be met—are incompetition with second gesture recognition criteria for a secondgesture—which are dependent on the contact(s) reaching the respectiveintensity threshold. In such competitions, the gesture is, optionally,not recognized as meeting the first gesture recognition criteria for thefirst gesture if the second gesture recognition criteria for the secondgesture are met first. For example, if a contact reaches the respectiveintensity threshold before the contact moves by a predefined amount ofmovement, a deep press gesture is detected rather than a swipe gesture.Conversely, if the contact moves by the predefined amount of movementbefore the contact reaches the respective intensity threshold, a swipegesture is detected rather than a deep press gesture. Even in suchcircumstances, the first gesture recognition criteria for the firstgesture still do not require that the intensity of the contact(s) meet arespective intensity threshold in order for the first gesturerecognition criteria to be met because if the contact stayed below therespective intensity threshold until an end of the gesture (e.g., aswipe gesture with a contact that does not increase to an intensityabove the respective intensity threshold), the gesture would have beenrecognized by the first gesture recognition criteria as a swipe gesture.As such, particular gesture recognition criteria that do not requirethat the intensity of the contact(s) meet a respective intensitythreshold in order for the particular gesture recognition criteria to bemet will (A) in some circumstances ignore the intensity of the contactwith respect to the intensity threshold (e.g. for a tap gesture) and/or(B) in some circumstances still be dependent on the intensity of thecontact with respect to the intensity threshold in the sense that theparticular gesture recognition criteria (e.g., for a long press gesture)will fail if a competing set of intensity-dependent gesture recognitioncriteria (e.g., for a deep press gesture) recognize an input ascorresponding to an intensity-dependent gesture before the particulargesture recognition criteria recognize a gesture corresponding to theinput (e.g., for a long press gesture that is competing with a deeppress gesture for recognition).

Graphics module 132 includes various known software components forrendering and displaying graphics on touch-sensitive display system 112or other display, including components for changing the visual impact(e.g., brightness, transparency, saturation, contrast or other visualproperty) of graphics that are displayed. As used herein, the term“graphics” includes any object that can be displayed to a user,including without limitation text, web pages, icons (such asuser-interface objects including soft keys), digital images, videos,animations and the like.

In some embodiments, graphics module 132 stores data representinggraphics to be used. Each graphic is, optionally, assigned acorresponding code. Graphics module 132 receives, from applicationsetc., one or more codes specifying graphics to be displayed along with,if necessary, coordinate data and other graphic property data, and thengenerates screen image data to output to display controller 156.

Haptic feedback module 133 includes various software components forgenerating instructions (e.g., instructions used by haptic feedbackcontroller 161) to produce tactile outputs using tactile outputgenerator(s) 167 at one or more locations on device 100 in response touser interactions with device 100.

Text input module 134, which is, optionally, a component of graphicsmodule 132, provides soft keyboards for entering text in variousapplications (e.g., contacts 137, e-mail 140, IM 141, browser 147, andany other application that needs text input).

GPS module 135 determines the location of the device and provides thisinformation for use in various applications (e.g., to telephone 138 foruse in location-based dialing, to camera 143 as picture/video metadata,and to applications that provide location-based services such as weatherwidgets, local yellow page widgets, and map/navigation widgets).

Applications 136 optionally include the following modules (or sets ofinstructions), or a subset or superset thereof:

-   -   contacts module 137 (sometimes called an address book or contact        list);    -   telephone module 138;    -   video conferencing module 139;    -   e-mail client module 140;    -   instant messaging (IM) module 141;    -   workout support module 142;    -   camera module 143 for still and/or video images;    -   image management module 144;    -   browser module 147;    -   calendar module 148;    -   widget modules 149, which optionally include one or more of:        weather widget 149-1, stocks widget 149-2, calculator widget        149-3, alarm clock widget 149-4, dictionary widget 149-5, and        other widgets obtained by the user, as well as user-created        widgets 149-6;    -   widget creator module 150 for making user-created widgets 149-6;    -   search module 151;    -   video and music player module 152, which is, optionally, made up        of a video player module and a music player module;    -   notes module 153;    -   map module 154; and/or    -   online video module 155.

Examples of other applications 136 that are, optionally, stored inmemory 102 include other word processing applications, other imageediting applications, drawing applications, presentation applications,JAVA-enabled applications, encryption, digital rights management, voicerecognition, and voice replication.

In conjunction with touch-sensitive display system 112, displaycontroller 156, contact module 130, graphics module 132, and text inputmodule 134, contacts module 137 includes executable instructions tomanage an address book or contact list (e.g., stored in applicationinternal state 192 of contacts module 137 in memory 102 or memory 370),including: adding name(s) to the address book; deleting name(s) from theaddress book; associating telephone number(s), e-mail address(es),physical address(es) or other information with a name; associating animage with a name; categorizing and sorting names; providing telephonenumbers and/or e-mail addresses to initiate and/or facilitatecommunications by telephone 138, video conference 139, e-mail 140, or IM141; and so forth.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111,microphone 113, touch-sensitive display system 112, display controller156, contact module 130, graphics module 132, and text input module 134,telephone module 138 includes executable instructions to enter asequence of characters corresponding to a telephone number, access oneor more telephone numbers in address book 137, modify a telephone numberthat has been entered, dial a respective telephone number, conduct aconversation and disconnect or hang up when the conversation iscompleted. As noted above, the wireless communication optionally usesany of a plurality of communications standards, protocols andtechnologies.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111,microphone 113, touch-sensitive display system 112, display controller156, optical sensor(s) 164, optical sensor controller 158, contactmodule 130, graphics module 132, text input module 134, contact list137, and telephone module 138, videoconferencing module 139 includesexecutable instructions to initiate, conduct, and terminate a videoconference between a user and one or more other participants inaccordance with user instructions.

In conjunction with RF circuitry 108, touch-sensitive display system112, display controller 156, contact module 130, graphics module 132,and text input module 134, e-mail client module 140 includes executableinstructions to create, send, receive, and manage e-mail in response touser instructions. In conjunction with image management module 144,e-mail client module 140 makes it very easy to create and send e-mailswith still or video images taken with camera module 143.

In conjunction with RF circuitry 108, touch-sensitive display system112, display controller 156, contact module 130, graphics module 132,and text input module 134, the instant messaging module 141 includesexecutable instructions to enter a sequence of characters correspondingto an instant message, to modify previously entered characters, totransmit a respective instant message (for example, using a ShortMessage Service (SMS) or Multimedia Message Service (MMS) protocol fortelephony-based instant messages or using XMPP, SIMPLE, Apple PushNotification Service (APNs) or IMPS for Internet-based instantmessages), to receive instant messages, and to view received instantmessages. In some embodiments, transmitted and/or received instantmessages optionally include graphics, photos, audio files, video filesand/or other attachments as are supported in an MMS and/or an EnhancedMessaging Service (EMS). As used herein, “instant messaging” refers toboth telephony-based messages (e.g., messages sent using SMS or MMS) andInternet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs,or IMPS).

In conjunction with RF circuitry 108, touch-sensitive display system112, display controller 156, contact module 130, graphics module 132,text input module 134, GPS module 135, map module 154, and video andmusic player module 152, workout support module 142 includes executableinstructions to create workouts (e.g., with time, distance, and/orcalorie burning goals); communicate with workout sensors (in sportsdevices and smart watches); receive workout sensor data; calibratesensors used to monitor a workout; select and play music for a workout;and display, store and transmit workout data.

In conjunction with touch-sensitive display system 112, displaycontroller 156, optical sensor(s) 164, optical sensor controller 158,contact module 130, graphics module 132, and image management module144, camera module 143 includes executable instructions to capture stillimages or video (including a video stream) and store them into memory102, modify characteristics of a still image or video, and/or delete astill image or video from memory 102.

In conjunction with touch-sensitive display system 112, displaycontroller 156, contact module 130, graphics module 132, text inputmodule 134, and camera module 143, image management module 144 includesexecutable instructions to arrange, modify (e.g., edit), or otherwisemanipulate, label, delete, present (e.g., in a digital slide show oralbum), and store still and/or video images.

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, and text input module 134, browser module 147 includes executableinstructions to browse the Internet in accordance with userinstructions, including searching, linking to, receiving, and displayingweb pages or portions thereof, as well as attachments and other fileslinked to web pages.

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, text input module 134, e-mail client module 140, and browser module147, calendar module 148 includes executable instructions to create,display, modify, and store calendars and data associated with calendars(e.g., calendar entries, to do lists, etc.) in accordance with userinstructions.

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, text input module 134, and browser module 147, widget modules 149are mini-applications that are, optionally, downloaded and used by auser (e.g., weather widget 149-1, stocks widget 149-2, calculator widget149-3, alarm clock widget 149-4, and dictionary widget 149-5) or createdby the user (e.g., user-created widget 149-6). In some embodiments, awidget includes an HTML (Hypertext Markup Language) file, a CSS(Cascading Style Sheets) file, and a JavaScript file. In someembodiments, a widget includes an XML (Extensible Markup Language) fileand a JavaScript file (e.g., Yahoo! Widgets).

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, text input module 134, and browser module 147, the widget creatormodule 150 includes executable instructions to create widgets (e.g.,turning a user-specified portion of a web page into a widget).

In conjunction with touch-sensitive display system 112, display systemcontroller 156, contact module 130, graphics module 132, and text inputmodule 134, search module 151 includes executable instructions to searchfor text, music, sound, image, video, and/or other files in memory 102that match one or more search criteria (e.g., one or more user-specifiedsearch terms) in accordance with user instructions.

In conjunction with touch-sensitive display system 112, display systemcontroller 156, contact module 130, graphics module 132, audio circuitry110, speaker 111, RF circuitry 108, and browser module 147, video andmusic player module 152 includes executable instructions that allow theuser to download and play back recorded music and other sound filesstored in one or more file formats, such as MP3 or AAC files, andexecutable instructions to display, present or otherwise play backvideos (e.g., on touch-sensitive display system 112, or on an externaldisplay connected wirelessly or via external port 124). In someembodiments, device 100 optionally includes the functionality of an MP3player, such as an iPod (trademark of Apple Inc.).

In conjunction with touch-sensitive display system 112, displaycontroller 156, contact module 130, graphics module 132, and text inputmodule 134, notes module 153 includes executable instructions to createand manage notes, to do lists, and the like in accordance with userinstructions.

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, text input module 134, GPS module 135, and browser module 147, mapmodule 154 includes executable instructions to receive, display, modify,and store maps and data associated with maps (e.g., driving directions;data on stores and other points of interest at or near a particularlocation; and other location-based data) in accordance with userinstructions.

In conjunction with touch-sensitive display system 112, display systemcontroller 156, contact module 130, graphics module 132, audio circuitry110, speaker 111, RF circuitry 108, text input module 134, e-mail clientmodule 140, and browser module 147, online video module 155 includesexecutable instructions that allow the user to access, browse, receive(e.g., by streaming and/or download), play back (e.g., on the touchscreen 112, or on an external display connected wirelessly or viaexternal port 124), send an e-mail with a link to a particular onlinevideo, and otherwise manage online videos in one or more file formats,such as H.264. In some embodiments, instant messaging module 141, ratherthan e-mail client module 140, is used to send a link to a particularonline video.

Each of the above identified modules and applications correspond to aset of executable instructions for performing one or more functionsdescribed above and the methods described in this application (e.g., thecomputer-implemented methods and other information processing methodsdescribed herein). These modules (i.e., sets of instructions) need notbe implemented as separate software programs, procedures or modules, andthus various subsets of these modules are, optionally, combined orotherwise re-arranged in various embodiments. In some embodiments,memory 102 optionally stores a subset of the modules and data structuresidentified above. Furthermore, memory 102 optionally stores additionalmodules and data structures not described above.

In some embodiments, device 100 is a device where operation of apredefined set of functions on the device is performed exclusivelythrough a touch screen and/or a touchpad. By using a touch screen and/ora touchpad as the primary input control device for operation of device100, the number of physical input control devices (such as push buttons,dials, and the like) on device 100 is, optionally, reduced.

The predefined set of functions that are performed exclusively through atouch screen and/or a touchpad optionally include navigation betweenuser interfaces. In some embodiments, the touchpad, when touched by theuser, navigates device 100 to a main, home, or root menu from any userinterface that is displayed on device 100. In such embodiments, a “menubutton” is implemented using a touchpad. In some other embodiments, themenu button is a physical push button or other physical input controldevice instead of a touchpad.

FIG. 1B is a block diagram illustrating example components for eventhandling in accordance with some embodiments. In some embodiments,memory 102 (in FIG. 1A) or 370 (FIG. 3) includes event sorter 170 (e.g.,in operating system 126) and a respective application 136-1 (e.g., anyof the aforementioned applications 136, 137-155, 380-390).

Event sorter 170 receives event information and determines theapplication 136-1 and application view 191 of application 136-1 to whichto deliver the event information. Event sorter 170 includes eventmonitor 171 and event dispatcher module 174. In some embodiments,application 136-1 includes application internal state 192, whichindicates the current application view(s) displayed on touch-sensitivedisplay system 112 when the application is active or executing. In someembodiments, device/global internal state 157 is used by event sorter170 to determine which application(s) is (are) currently active, andapplication internal state 192 is used by event sorter 170 to determineapplication views 191 to which to deliver event information.

In some embodiments, application internal state 192 includes additionalinformation, such as one or more of: resume information to be used whenapplication 136-1 resumes execution, user interface state informationthat indicates information being displayed or that is ready for displayby application 136-1, a state queue for enabling the user to go back toa prior state or view of application 136-1, and a redo/undo queue ofprevious actions taken by the user.

Event monitor 171 receives event information from peripherals interface118. Event information includes information about a sub-event (e.g., auser touch on touch-sensitive display system 112, as part of amulti-touch gesture). Peripherals interface 118 transmits information itreceives from I/O subsystem 106 or a sensor, such as proximity sensor166, accelerometer(s) 168, and/or microphone 113 (through audiocircuitry 110). Information that peripherals interface 118 receives fromI/O subsystem 106 includes information from touch-sensitive displaysystem 112 or a touch-sensitive surface.

In some embodiments, event monitor 171 sends requests to the peripheralsinterface 118 at predetermined intervals. In response, peripheralsinterface 118 transmits event information. In other embodiments,peripheral interface 118 transmits event information only when there isa significant event (e.g., receiving an input above a predeterminednoise threshold and/or for more than a predetermined duration).

In some embodiments, event sorter 170 also includes a hit viewdetermination module 172 and/or an active event recognizer determinationmodule 173.

Hit view determination module 172 provides software procedures fordetermining where a sub-event has taken place within one or more views,when touch-sensitive display system 112 displays more than one view.Views are made up of controls and other elements that a user can see onthe display.

Another aspect of the user interface associated with an application is aset of views, sometimes herein called application views or userinterface windows, in which information is displayed and touch-basedgestures occur. The application views (of a respective application) inwhich a touch is detected optionally correspond to programmatic levelswithin a programmatic or view hierarchy of the application. For example,the lowest level view in which a touch is detected is, optionally,called the hit view, and the set of events that are recognized as properinputs are, optionally, determined based, at least in part, on the hitview of the initial touch that begins a touch-based gesture.

Hit view determination module 172 receives information related tosub-events of a touch-based gesture. When an application has multipleviews organized in a hierarchy, hit view determination module 172identifies a hit view as the lowest view in the hierarchy which shouldhandle the sub-event. In most circumstances, the hit view is the lowestlevel view in which an initiating sub-event occurs (i.e., the firstsub-event in the sequence of sub-events that form an event or potentialevent). Once the hit view is identified by the hit view determinationmodule, the hit view typically receives all sub-events related to thesame touch or input source for which it was identified as the hit view.

Active event recognizer determination module 173 determines which viewor views within a view hierarchy should receive a particular sequence ofsub-events. In some embodiments, active event recognizer determinationmodule 173 determines that only the hit view should receive a particularsequence of sub-events. In other embodiments, active event recognizerdetermination module 173 determines that all views that include thephysical location of a sub-event are actively involved views, andtherefore determines that all actively involved views should receive aparticular sequence of sub-events. In other embodiments, even if touchsub-events were entirely confined to the area associated with oneparticular view, views higher in the hierarchy would still remain asactively involved views.

Event dispatcher module 174 dispatches the event information to an eventrecognizer (e.g., event recognizer 180). In embodiments including activeevent recognizer determination module 173, event dispatcher module 174delivers the event information to an event recognizer determined byactive event recognizer determination module 173. In some embodiments,event dispatcher module 174 stores in an event queue the eventinformation, which is retrieved by a respective event receiver module182.

In some embodiments, operating system 126 includes event sorter 170.Alternatively, application 136-1 includes event sorter 170. In yet otherembodiments, event sorter 170 is a stand-alone module, or a part ofanother module stored in memory 102, such as contact/motion module 130.

In some embodiments, application 136-1 includes a plurality of eventhandlers 190 and one or more application views 191, each of whichincludes instructions for handling touch events that occur within arespective view of the application's user interface. Each applicationview 191 of the application 136-1 includes one or more event recognizers180. Typically, a respective application view 191 includes a pluralityof event recognizers 180. In other embodiments, one or more of eventrecognizers 180 are part of a separate module, such as a user interfacekit (not shown) or a higher level object from which application 136-1inherits methods and other properties. In some embodiments, a respectiveevent handler 190 includes one or more of: data updater 176, objectupdater 177, GUI updater 178, and/or event data 179 received from eventsorter 170. Event handler 190 optionally utilizes or calls data updater176, object updater 177 or GUI updater 178 to update the applicationinternal state 192. Alternatively, one or more of the application views191 includes one or more respective event handlers 190. Also, in someembodiments, one or more of data updater 176, object updater 177, andGUI updater 178 are included in a respective application view 191.

A respective event recognizer 180 receives event information (e.g.,event data 179) from event sorter 170, and identifies an event from theevent information. Event recognizer 180 includes event receiver 182 andevent comparator 184. In some embodiments, event recognizer 180 alsoincludes at least a subset of: metadata 183, and event deliveryinstructions 188 (which optionally include sub-event deliveryinstructions).

Event receiver 182 receives event information from event sorter 170. Theevent information includes information about a sub-event, for example, atouch or a touch movement. Depending on the sub-event, the eventinformation also includes additional information, such as location ofthe sub-event. When the sub-event concerns motion of a touch, the eventinformation optionally also includes speed and direction of thesub-event. In some embodiments, events include rotation of the devicefrom one orientation to another (e.g., from a portrait orientation to alandscape orientation, or vice versa), and the event informationincludes corresponding information about the current orientation (alsocalled device attitude) of the device.

Event comparator 184 compares the event information to predefined eventor sub-event definitions and, based on the comparison, determines anevent or sub-event, or determines or updates the state of an event orsub-event. In some embodiments, event comparator 184 includes eventdefinitions 186. Event definitions 186 contain definitions of events(e.g., predefined sequences of sub-events), for example, event 1(187-1), event 2 (187-2), and others. In some embodiments, sub-events inan event 187 include, for example, touch begin, touch end, touchmovement, touch cancellation, and multiple touching. In one example, thedefinition for event 1 (187-1) is a double tap on a displayed object.The double tap, for example, comprises a first touch (touch begin) onthe displayed object for a predetermined phase, a first lift-off (touchend) for a predetermined phase, a second touch (touch begin) on thedisplayed object for a predetermined phase, and a second lift-off (touchend) for a predetermined phase. In another example, the definition forevent 2 (187-2) is a dragging on a displayed object. The dragging, forexample, comprises a touch (or contact) on the displayed object for apredetermined phase, a movement of the touch across touch-sensitivedisplay system 112, and lift-off of the touch (touch end). In someembodiments, the event also includes information for one or moreassociated event handlers 190.

In some embodiments, event definition 187 includes a definition of anevent for a respective user-interface object. In some embodiments, eventcomparator 184 performs a hit test to determine which user-interfaceobject is associated with a sub-event. For example, in an applicationview in which three user-interface objects are displayed ontouch-sensitive display system 112, when a touch is detected ontouch-sensitive display system 112, event comparator 184 performs a hittest to determine which of the three user-interface objects isassociated with the touch (sub-event). If each displayed object isassociated with a respective event handler 190, the event comparatoruses the result of the hit test to determine which event handler 190should be activated. For example, event comparator 184 selects an eventhandler associated with the sub-event and the object triggering the hittest.

In some embodiments, the definition for a respective event 187 alsoincludes delayed actions that delay delivery of the event informationuntil after it has been determined whether the sequence of sub-eventsdoes or does not correspond to the event recognizer's event type.

When a respective event recognizer 180 determines that the series ofsub-events do not match any of the events in event definitions 186, therespective event recognizer 180 enters an event impossible, eventfailed, or event ended state, after which it disregards subsequentsub-events of the touch-based gesture. In this situation, other eventrecognizers, if any, that remain active for the hit view continue totrack and process sub-events of an ongoing touch-based gesture.

In some embodiments, a respective event recognizer 180 includes metadata183 with configurable properties, flags, and/or lists that indicate howthe event delivery system should perform sub-event delivery to activelyinvolved event recognizers. In some embodiments, metadata 183 includesconfigurable properties, flags, and/or lists that indicate how eventrecognizers interact, or are enabled to interact, with one another. Insome embodiments, metadata 183 includes configurable properties, flags,and/or lists that indicate whether sub-events are delivered to varyinglevels in the view or programmatic hierarchy.

In some embodiments, a respective event recognizer 180 activates eventhandler 190 associated with an event when one or more particularsub-events of an event are recognized. In some embodiments, a respectiveevent recognizer 180 delivers event information associated with theevent to event handler 190. Activating an event handler 190 is distinctfrom sending (and deferred sending) sub-events to a respective hit view.In some embodiments, event recognizer 180 throws a flag associated withthe recognized event, and event handler 190 associated with the flagcatches the flag and performs a predefined process.

In some embodiments, event delivery instructions 188 include sub-eventdelivery instructions that deliver event information about a sub-eventwithout activating an event handler. Instead, the sub-event deliveryinstructions deliver event information to event handlers associated withthe series of sub-events or to actively involved views. Event handlersassociated with the series of sub-events or with actively involved viewsreceive the event information and perform a predetermined process.

In some embodiments, data updater 176 creates and updates data used inapplication 136-1. For example, data updater 176 updates the telephonenumber used in contacts module 137, or stores a video file used in videoand music player module 152. In some embodiments, object updater 177creates and updates objects used in application 136-1. For example,object updater 177 creates a new user-interface object or updates theposition of a user-interface object. GUI updater 178 updates the GUI.For example, GUI updater 178 prepares display information and sends itto graphics module 132 for display on a touch-sensitive display.

In some embodiments, event handler(s) 190 includes or has access to dataupdater 176, object updater 177, and GUI updater 178. In someembodiments, data updater 176, object updater 177, and GUI updater 178are included in a single module of a respective application 136-1 orapplication view 191. In other embodiments, they are included in two ormore software modules.

It shall be understood that the foregoing discussion regarding eventhandling of user touches on touch-sensitive displays also applies toother forms of user inputs to operate multifunction devices 100 withinput-devices, not all of which are initiated on touch screens. Forexample, mouse movement and mouse button presses, optionally coordinatedwith single or multiple keyboard presses or holds; contact movementssuch as taps, drags, scrolls, etc., on touch-pads; pen stylus inputs;movement of the device; oral instructions; detected eye movements;biometric inputs; and/or any combination thereof are optionally utilizedas inputs corresponding to sub-events which define an event to berecognized.

FIG. 2 illustrates a portable multifunction device 100 having a touchscreen (e.g., touch-sensitive display system 112, FIG. 1A) in accordancewith some embodiments. The touch screen optionally displays one or moregraphics within user interface (UI) 200. In these embodiments, as wellas others described below, a user is enabled to select one or more ofthe graphics by making a gesture on the graphics, for example, with oneor more fingers 202 (not drawn to scale in the figure) or one or morestyluses 203 (not drawn to scale in the figure). In some embodiments,selection of one or more graphics occurs when the user breaks contactwith the one or more graphics. In some embodiments, the gestureoptionally includes one or more taps, one or more swipes (from left toright, right to left, upward and/or downward) and/or a rolling of afinger (from right to left, left to right, upward and/or downward) thathas made contact with device 100. In some implementations orcircumstances, inadvertent contact with a graphic does not select thegraphic. For example, a swipe gesture that sweeps over an applicationicon optionally does not select the corresponding application when thegesture corresponding to selection is a tap.

Device 100 optionally also includes one or more physical buttons, suchas “home” or menu button 204. As described previously, menu button 204is, optionally, used to navigate to any application 136 in a set ofapplications that are, optionally executed on device 100. Alternatively,in some embodiments, the menu button is implemented as a soft key in aGUI displayed on the touch-screen display.

In some embodiments, device 100 includes the touch-screen display, menubutton 204 (sometimes called home button 204), push button 206 forpowering the device on/off and locking the device, volume adjustmentbutton(s) 208, Subscriber Identity Module (SIM) card slot 210, head setjack 212, and docking/charging external port 124. Push button 206 is,optionally, used to turn the power on/off on the device by depressingthe button and holding the button in the depressed state for apredefined time interval; to lock the device by depressing the buttonand releasing the button before the predefined time interval haselapsed; and/or to unlock the device or initiate an unlock process. Insome embodiments, device 100 also accepts verbal input for activation ordeactivation of some functions through microphone 113. Device 100 also,optionally, includes one or more contact intensity sensors 165 fordetecting intensities of contacts on touch-sensitive display system 112and/or one or more tactile output generators 167 for generating tactileoutputs for a user of device 100.

FIG. 3A is a block diagram of an example multifunction device with adisplay and a touch-sensitive surface in accordance with someembodiments. Device 300 need not be portable. In some embodiments,device 300 is a laptop computer, a desktop computer, a tablet computer,a multimedia player device, a navigation device, an educational device(such as a child's learning toy), a gaming system, or a control device(e.g., a home or industrial controller). Device 300 typically includesone or more processing units (CPU's) 310, one or more network or othercommunications interfaces 360, memory 370, and one or more communicationbuses 320 for interconnecting these components. Communication buses 320optionally include circuitry (sometimes called a chipset) thatinterconnects and controls communications between system components.Device 300 includes input/output (I/O) interface 330 comprising display340, which is typically a touch-screen display. In some embodiments,display 340 is referred to as a display generation component. I/Ointerface 330 also optionally includes a keyboard and/or mouse (or otherpointing device) 350 and touchpad 355, tactile output generator 357 forgenerating tactile outputs on device 300 (e.g., similar to tactileoutput generator(s) 167 described above with reference to FIG. 1A),sensors 359 (e.g., optical, acceleration, proximity, touch-sensitive,and/or contact intensity sensors similar to contact intensity sensor(s)165 described above with reference to FIG. 1A). Memory 370 includeshigh-speed random access memory, such as DRAM, SRAM, DDR RAM or otherrandom access solid state memory devices; and optionally includesnon-volatile memory, such as one or more magnetic disk storage devices,optical disk storage devices, flash memory devices, or othernon-volatile solid state storage devices. Memory 370 optionally includesone or more storage devices remotely located from CPU(s) 310. In someembodiments, memory 370 stores programs, modules, and data structuresanalogous to the programs, modules, and data structures stored in memory102 of portable multifunction device 100 (FIG. 1A), or a subset thereof.Furthermore, memory 370 optionally stores additional programs, modules,and data structures not present in memory 102 of portable multifunctiondevice 100. For example, memory 370 of device 300 optionally storesdrawing module 380, presentation module 382, word processing module 384,website creation module 386, disk authoring module 388, and/orspreadsheet module 390, while memory 102 of portable multifunctiondevice 100 (FIG. 1A) optionally does not store these modules.

Each of the above identified elements in FIG. 3A are, optionally, storedin one or more of the previously mentioned memory devices. Each of theabove identified modules corresponds to a set of instructions forperforming a function described above. The above identified modules orprograms (i.e., sets of instructions) need not be implemented asseparate software programs, procedures or modules, and thus varioussubsets of these modules are, optionally, combined or otherwisere-arranged in various embodiments. In some embodiments, memory 370optionally stores a subset of the modules and data structures identifiedabove. Furthermore, memory 370 optionally stores additional modules anddata structures not described above.

FIG. 3B is a block diagram of an example wearable audio output device301 in accordance with some embodiments. In some embodiments, wearableaudio output device 301 is one or more in-ear earphone(s), earbud(s),over-ear headphone(s), or the like. In some examples, wearable audiooutput device 301 is a single earphone or earbud. In some examples,wearable audio output device 301 includes a pair of earphones or earbuds(e.g., one for each of a user's ears). In some examples, wearable audiooutput device 301 includes over-ear headphones (e.g., headphones withtwo over-ear earcups to be placed over a user's ears and optionallyconnected by a headband). In some embodiments, wearable audio outputdevice 301 includes one or more speakers 306 for providing audio output(e.g., to a user's ear). In some embodiments, wearable audio outputdevice 301 includes one or more pose sensors 304 (e.g., includingaccelerometer(s) and/or attitude sensor(s)) to detect a pose (e.g.,position and/or orientation) and changes in the pose of wearable audiooutput device 301, or of a user (e.g., a wearer) of wearable audiooutput device 301, relative to a physical environment. In someembodiments, wearable audio output device 301 conditionally outputsaudio based on a pose of wearable audio output device 301 as determinedby pose sensor(s) 304. In some embodiments, audio I/O logic 312determines the pose of wearable audio output device 301 or of the wearerof wearable audio output device 301, and, in some embodiments, audio I/Ologic 312 controls the resulting conditional outputting of audio.

In some embodiments, wearable audio output device 301 includes one ormore microphones 302 for receiving audio input. In some embodiments,microphone(s) 302 detect speech from a user wearing wearable audiooutput device 301 and/or ambient noise around wearable audio outputdevice 301. In some embodiments, as described in more detail herein withreference to FIG. 3C, multiple microphones of microphones 302 arepositioned at different locations on wearable audio output device 301 tomeasure speech and/or ambient noise at different locations aroundwearable audio output device 301. In some embodiments, audio I/O logic312 detects or recognizes speech or ambient noise based on informationreceived from microphone(s) 302.

In some embodiments, wearable audio output device 301 includes one ormore other input devices 308, such as a touch-sensitive surface (fordetecting touch inputs), one or more placement sensors to detectpositioning or placement of wearable audio output device 301 relative toa user's ear (such as to detect placement of wearable audio outputdevice 301 in or on a user's ear), and/or other input device by which auser can interact with and provide inputs to wearable audio outputdevice 301. In some embodiments, inputs provided via input device(s) 308are processed by audio I/O logic 312. In some embodiments, audio I/Ologic 312 is in communication with a separate device (e.g., device 100,FIG. 1A, or device 300, FIG. 3A) that provides instructions or contentfor audio output, and that optionally receives and processes inputs (orinformation about inputs) provided via microphone(s) 302, pose sensor(s)304, and/or input device(s) 308, or via one or more input devices of theseparate device. In some embodiments, audio I/O logic 312 is located indevice 100 (e.g., as part of peripherals interface 118, FIG. 1A) ordevice 300 (e.g., as part of I/O interface 330, FIG. 3A), instead ofdevice 301, or alternatively is located in part in device 100 and inpart in device 301, or in part in device 300 and in part in device 301.

FIG. 3C illustrates example audio control by a wearable audio outputdevice in accordance with some embodiments. In some embodiments, when awearable audio output device having over-ear earcups is worn over auser's ear, the earcups act as physical barriers that block at leastsome ambient sound from the surrounding physical environment fromreaching the user's ear. For example, in FIG. 3C, wearable audio outputdevice 301 is worn by a user such that earcup 314 is over the user'sleft ear. In some embodiments, a first microphone (or, in someembodiments, a first set of one or more microphones) 302-1 (e.g., ofmicrophones 302, FIG. 3B) is located on wearable audio output device 301so as to detect ambient sound, represented by waveform 322, in region316 of a physical environment surrounding (e.g., outside of) earcup 314.In some embodiments, earcup 314 blocks some, but not necessarily all, ofthe ambient sound in the surrounding physical environment from reachingthe user's ear. In some embodiments, a second microphone (or, in someembodiments, a second set of one or more microphones) 302-2 (e.g., ofmicrophones 302, FIG. 3B) is located on wearable audio output device 301so as to detect any ambient sound, represented by waveform 324, that isnot completely blocked by earcup 314 and that can be heard in region 318inside earcup 314. Accordingly, in some circumstances in which wearableaudio output device 301 is not producing a noise-cancelling (also called“antiphase”) audio signal to cancel (e.g., attenuate) ambient sound fromthe surrounding physical environment, as indicated by waveform 326-1,ambient sound waveform 324 is perceivable by the user, as indicated bywaveform 328-1. In some circumstances in which wearable audio outputdevice 301 is producing an antiphase audio signal to cancel ambientsound, as indicated by waveform 326-2, ambient sound waveform 324 is notperceivable by the user, as indicated by waveform 328-2.

In some embodiments, ambient sound waveform 322 is compared toattenuated ambient sound waveform 324 (e.g., by wearable audio outputdevice 301 or a component of wearable audio output device 301, such asaudio I/O logic 312, or by an electronic device that is in communicationwith wearable audio output device 301) to determine the passiveattenuation provided by wearable audio output device 301. In someembodiments, the amount of passive attenuation provided by wearableaudio output device 301 is taken into account when providing theantiphase audio signal to cancel ambient sound from the surroundingphysical environment. For example, antiphase audio signal waveform 326-2is configured to cancel attenuated ambient sound waveform 324 ratherthan unattenuated ambient sound waveform 322.

Attention is now directed towards embodiments of user interfaces (“UI”)that are, optionally, implemented on portable multifunction device 100.

FIG. 4A illustrates an example user interface for a menu of applicationson portable multifunction device 100 in accordance with someembodiments. Similar user interfaces are, optionally, implemented ondevice 300. In some embodiments, user interface 400 includes thefollowing elements, or a subset or superset thereof:

-   -   Signal strength indicator(s) for wireless communication(s), such        as cellular and Wi-Fi signals;    -   Time;    -   a Bluetooth indicator;    -   a Battery status indicator;    -   Tray 408 with icons for frequently used applications, such as:        -   Icon 416 for telephone module 138, labeled “Phone,” which            optionally includes an indicator 414 of the number of missed            calls or voicemail messages;        -   Icon 418 for e-mail client module 140, labeled “Mail,” which            optionally includes an indicator 410 of the number of unread            e-mails;        -   Icon 420 for browser module 147, labeled “Browser;” and        -   Icon 422 for video and music player module 152, labeled            “Music;” and    -   Icons for other applications, such as:        -   Icon 424 for IM module 141, labeled “Messages;”        -   Icon 426 for calendar module 148, labeled “Calendar;”        -   Icon 428 for image management module 144, labeled “Photos;”        -   Icon 430 for camera module 143, labeled “Camera;”        -   Icon 432 for online video module 155, labeled “Online            Video;”        -   Icon 434 for stocks widget 149-2, labeled “Stocks;”        -   Icon 436 for map module 154, labeled “Maps;”        -   Icon 438 for weather widget 149-1, labeled “Weather;”        -   Icon 440 for alarm clock widget 149-4, labeled “Clock;”        -   Icon 442 for workout support module 142, labeled “Workout            Support;”        -   Icon 444 for notes module 153, labeled “Notes;” and        -   Icon 446 for a settings application or module, which            provides access to settings for device 100 and its various            applications 136.

It should be noted that the icon labels illustrated in FIG. 4A aremerely examples. For example, other labels are, optionally, used forvarious application icons. In some embodiments, a label for a respectiveapplication icon includes a name of an application corresponding to therespective application icon. In some embodiments, a label for aparticular application icon is distinct from a name of an applicationcorresponding to the particular application icon.

FIG. 4B illustrates an example user interface on a device (e.g., device300, FIG. 3) with a touch-sensitive surface 451 (e.g., a tablet ortouchpad 355, FIG. 3) that is separate from the display 450. Although insome examples, inputs may be received on touch screen display 112 (wherethe touch sensitive surface and the display are combined), in someembodiments, the device detects inputs on a touch-sensitive surface thatis separate from the display, as shown in FIG. 4B. In some embodiments,the touch-sensitive surface (e.g., 451 in FIG. 4B) has a primary axis(e.g., 452 in FIG. 4B) that corresponds to a primary axis (e.g., 453 inFIG. 4B) on the display (e.g., 450). In accordance with theseembodiments, the device detects contacts (e.g., 460 and 462 in FIG. 4B)with the touch-sensitive surface 451 at locations that correspond torespective locations on the display (e.g., in FIG. 4B, 460 correspondsto 468 and 462 corresponds to 470). In this way, user inputs (e.g.,contacts 460 and 462, and movements thereof) detected by the device onthe touch-sensitive surface (e.g., 451 in FIG. 4B) are used by thedevice to manipulate the user interface on the display (e.g., 450 inFIG. 4B) of the multifunction device when the touch-sensitive surface isseparate from the display. It should be understood that similar methodsare, optionally, used for other user interfaces described herein.

User Interfaces and Associated Processes

Attention is now directed towards embodiments of user interfaces (“UI”)and associated processes that may be implemented on an electronicdevice, such as portable multifunction device 100 or device 300, with adisplay, a touch-sensitive surface, (optionally) one or more tactileoutput generators for generating tactile outputs, and (optionally) oneor more sensors to detect intensities of contacts with thetouch-sensitive surface.

FIGS. 5A-5L illustrate example adaptive audio outputs in response toexample user interactions with wearable audio output devices inaccordance with some embodiments. FIGS. 6A-6J illustrate exampleadaptive audio outputs in response to example changes in the audioproperties of a surrounding physical environment in accordance with someembodiments. The audio outputs, user interactions, and ambient audiochanges in these figures are used to illustrate the processes describedbelow, including the processes in FIGS. 7A-7B and 8A-8B. For convenienceof explanation, some of the embodiments will be discussed with referenceto operations performed using a wearable audio output device that isworn by a user and that is in communication with and separate from anelectronic device having a touch-sensitive display system 112 or display340 that is separate from a touch-sensitive input device such astouchpad 355. In some embodiments, the operations are performed inresponse to instructions received by the wearable audio output devicefrom the electronic device, based on processing performed at theelectronic device. In some embodiments, the operations are performed bythe wearable audio output device based on processing performed at thewearable audio output device. However, in some cases, analogousoperations are optionally performed using audio output devices that arepart of a device having a display generation component and/ortouch-sensitive input device (e.g., a wearable device, such asheadphones or a headset, that integrates the one or more audio outputdevices with a display and/or touch-sensitive input device).

FIGS. 5A-5L illustrate example adaptive audio outputs in response toexample user interactions with wearable audio output devices, includingchanges in user pose, in accordance with some embodiments.

FIGS. 5A-5E illustrate changes in audio outputs provided by headphonesworn by a user in response to changes in the pose of the user betweensitting and standing positions. FIG. 5A illustrates user 502 seated andsitting upright, and wearing headphones 504, in physical environment500. Expanded view 501 illustrates headphones 504 playing music so thatthe music sounds as if it is coming from simulated spatial region 506surrounding user 502's head.

FIGS. 5B-5C illustrate a transition from FIG. 5A. In particular, FIG. 5Billustrates that user 502 has assumed a standing position. Expanded view501 in FIG. 5B illustrates that, in response to the change in the poseof user 502 from sitting (FIG. 5A) to standing (FIG. 5B), headphones 504move the simulated spatial location of the music being played so thatthe music sounds as if it is coming from simulated spatial location 508that is above user 502's head, which is distinct from simulated spatialregion 506 surrounding user 502's head, and decrease the volume at whichthe music is being played.

In some embodiments, in response to the change in the pose of user 502from sitting (FIG. 5A) to standing (FIG. 5B), headphones 504 fade musicplayback out completely. Accordingly, FIG. 5C illustrates an optionalfurther transition where, after the simulated spatial location of themusic is moved to above user 502's head and after the volume at whichthe music is being played is decreased (as shown in FIG. 5B anddescribed above), headphones 504 stop playing music entirely (e.g., themusic volume is decreased to zero or paused/stopped), as shown inexpanded view 501 in FIG. 5C.

In other words, in FIGS. 5B-5C, user 502 is provided with a lessimmersive and thus less obstructive audio experience in response tostanding up from a seated position (or, in other embodiments, inresponse to sitting or standing up from a reclining or laying downposition), to make it easier for user 502 to interact with hissurrounding physical environment, as the change in pose may indicateuser 502's intention to interact with his surrounding physicalenvironment. In contrast, in FIG. 5A, user 502 is provided with a moreimmersive audio experience while seated (or, in other embodiments, whilereclining or laying down), in which case user 502 may be less likely tointeract with his surrounding physical environment.

FIGS. 5D-5E illustrate a transition from FIGS. 5B-5C. In particular,FIG. 5D illustrates that user 502 has resumed the same seated pose as inFIG. 5A. In response to the change in pose of user 502 from standing(FIG. 5C) to sitting (FIG. 5D), headphones 504 resume playing music asin FIG. 5A. In some embodiments, as shown in FIGS. 5D-5E, headphones 504gradually resume music playback, for example by initially playing musicat simulated spatial location 508 above user 502's head (FIG. 5D) andlater moving the simulated spatial location of the music so that themusic sounds as if it is coming from simulated spatial region 506surrounding user 502's head (FIG. 5E). In addition, in some embodiments,headphones 504 gradually increase the music volume from zero to areduced volume initially relative to the original volume as in FIG. 5A(as shown in expanded view 501 in FIG. 5D) and later to the originalvolume as in FIG. 5A (as shown in expanded view 501 in FIG. 5E). In someembodiments, the simulated spatial location of the music is changedgradually in conjunction with gradual increases in the music volume. Insome embodiments where headphones 504 do not fade music playback outcompletely in response to a change in user pose (e.g., in embodimentswhere the scenario illustrated in FIG. 5C does not take place), user 502resuming a seated pose results in a transition from the scenario of FIG.5B to the scenario of FIG. 5E without first transitioning through theintermediate scenario of FIG. 5D.

FIGS. 5F-5G illustrate changes in audio outputs provided by headphones504 in response to a change in the pose of user 502 that includesmovement between different types of spaces. FIG. 5F illustrates user 502standing in public space 510. Expanded view 501 in FIG. 5F illustratesthat, while user 502 is standing in public space 510, headphones 504play music quietly, and so that the music sounds as if it is coming fromsimulated spatial location 508 that is above user 502's head. FIG. 5Gillustrates that user 502 has moved to private space 512. Expanded view501 in FIG. 5G illustrates that, in response to user 502 moving frompublic space 510 to private space 512, headphones 504 increase thevolume of music playback and move the simulated spatial location of themusic so that the music sounds as if it is coming from simulated spatialregion 506 surrounding user 502's head.

In other words, in FIG. 5F, user 502 is provided with a less immersiveand thus less obstructive audio experience while in a public space(e.g., public space 510), to make it easier for user 502 to interactwith his surrounding physical environment. In contrast, in FIG. 5G, user502 is provided with a more immersive audio experience while in aprivate space (e.g., private space 512), where user 502 may be lesslikely to interact with his surrounding physical environment.

FIG. 5H-5L illustrate changes in audio outputs provided by headphones504 worn by user 502, as well as changes in visual feedback provided ona display of an electronic device that is in communication withheadphones 504, in response to changes in the pose of user 502 betweenreclining and sitting positions. FIG. 5H illustrates user 502 recliningin physical environment 500. Expanded view 501 illustrates headphones504 playing music so that the music sounds as if it is coming fromsimulated spatial region 506 surrounding user 502's head. Expanded view503 illustrates a view of physical environment 500 from the perspectiveof user 502, including a view of device 514 and of a portion of plant518 that is visible behind device 514. Device 514 displays a full-screenuser interface for playing video 516. In the example shown in FIG. 5H,the music being played by headphones 504 is an audio track for the videocontent of video 516.

FIG. 5I illustrates that, as user 502 continues to watch video content516, a notification 520 is displayed on device 514. FIG. 5J illustratesthat, in response to notification 520, user 502 sits up, thus changingpose from reclining to sitting upright.

FIG. 5K illustrates a transition from FIG. 5J in accordance with someembodiments. In FIG. 5K, in response to the change in the pose of user502 from reclining to sitting upright, the levels of immersion withrespect to both video 516 displayed on device 514 and the music providedby headphones 504 are reduced. In particular, device 514 pauses playbackof video 516, reduces the scale of video 516 so that video 516 isdisplayed (e.g., while paused) in an inset window, and moves the insetwindow to a corner of the display of device 514. Device 514 alsodisplays live view 522 of physical environment 500 (e.g., from one ormore cameras of device 514). In addition, headphones 504 move thesimulated spatial location of the music so that the music sounds as ifit is coming from simulated spatial location 508 above user 502's head,and decrease the volume at which the music is being played. In someembodiments, headphones 504 fade the music playback out completely, asdescribed herein with reference to FIGS. 5B-5C.

FIG. 5L illustrates an alternate transition from FIG. 5J in accordancewith some embodiments. Like in FIG. 5K, in FIG. 5L, in response to thechange in the pose of user 502 from reclining to sitting upright, thelevels of immersion with respect to both video 516 displayed on device514 and the music provided by headphones 504 are reduced, in part bydevice 514 pausing playback of video 516 and by headphones 504 movingthe simulated spatial location of the music so that the music sounds asif it is coming from simulated spatial location 508 above user 502'shead, and decreasing the volume at which the music is being played.However, in FIG. 5L, instead of shrinking video 516 to an inset windowand moving video 516 to a corner of the display, device 514 increasesthe transparency of video 516 and displays at least a portion of liveview 522 of physical environment 500 (e.g., from one or more cameras ofdevice 514) so that the live view of physical environment 500 appears asif it is being viewed through a partially-transparent video 516. Inaddition, device 514 displays beacons 524-1 and 524-2 highlighting user502's wallet 526 and keys 528, respectively, to help user 502 respond tonotification 520 (e.g., to remind user 502 of items to take when leavingto attend the meeting indicated by notification 520).

FIGS. 6A-6J illustrate example adaptive audio outputs in response toexample changes in the audio properties of a surrounding physicalenvironment in accordance with some embodiments.

FIGS. 6A-6D illustrate changes in audio outputs provided by headphonesworn by a user in response to increase and decreases in ambient sound,and particularly speech, in a physical environment surrounding the user.FIG. 6A illustrates user 602 seated and wearing headphones 604 inphysical environment 600. Headphones 604 are in communication with anelectronic device (e.g., device 100, FIG. 1A) and provide audio outputcorresponding to audio content from the device, represented by devicecontent waveform 606. Ambient sound present in physical environment 600is represented by ambient sound waveform 608. Ambient sound fromphysical environment 600 that is not completely blocked by headphones604 (e.g., due to imperfect passive attenuation by headphones 604, asdescribed herein with reference to FIG. 3C) and that can be heard insidethe earcups of headphones 604 is represented by attenuated ambient soundwaveform 610. In the example illustrated in FIG. 6A, the ambient soundpresent in physical environment 600 is considered to satisfy predefinedaudio output criteria (e.g., audio output criteria associated withproviding device audio content, for example at a volume level selectedby user 602 or automatically determined by headphones 604 or by thedevice in communication with headphones 604). In addition, active noisecontrol is enabled for headphones 604, and thus a noise-cancellingsignal, represented by antiphase waveform 612, is provided to cancelattenuated ambient sound waveform 610. As a result of the active noisecontrol, no ambient sound is perceived by user 602, as indicated byperceived ambient sound waveform 614.

FIGS. 6B-6C illustrate a transition from FIG. 6A. In particular, FIG. 6Billustrates that another person 616 has approached user 602 and begunspeaking to user 602. The increase in the ambient sound in physicalenvironment 600 due to person 616 speaking is shown by ambient soundwaveform 608 and by attenuated ambient sound waveform 610. As a result,the ambient sound present in physical environment 600 is no longerconsidered to satisfy the predefined audio output criteria (e.g., audiooutput criteria associated with providing device audio content). In someembodiments, the ambient sound present in physical environment 600 inFIGS. 6B-6C is considered to satisfy different, second predefined audiooutput criteria (e.g., audio output criteria associated with a biastoward ambient sound). Initially after person 616 begins speaking,headphones 604 continue to output audio content from the device, asshown by device content waveform 606, and also continue to cancelambient sound, as shown by antiphase waveform 612. As a result, user 602continues to hear device content without hearing ambient sound.

FIG. 6C illustrates modification of the audio outputs provided byheadphones 604 in response to detecting the increase in ambient sound inphysical environment 600 (here, due to person 616 speaking) and inresponse to the ambient sound in physical environment 600 no longersatisfying the predefined audio output criteria. Device content waveform606 in FIG. 6C shows that the volume level of the audio content from thedevice is reduced relative to the level shown by device content waveform606 in preceding FIG. 6B. Also, active noise control is disabled, asshown by antiphase waveform 612 in FIG. 6C. In addition, headphones 604begin to actively pass through ambient sound from physical environment600 (e.g., using a microphone, such as microphone 302-1, FIG. 3C, toacquire ambient sound from physical environment 600 that is outside theearcups of headphones 604), so that user 602 hears ambient sound at avolume level above the attenuated ambient sound level resulting frompassive attenuation by headphones 604. In some embodiments, user 602hears the ambient sound at a volume level that is approximately equal to(e.g., within 10 percent, 15 percent or 20 percent of) the volume levelof the ambient sound in physical environment 600 (e.g., to reverse theattenuation by headphones 604, so that user 602 can hear and respond tosounds in physical environment 600 as if user 602 were not wearingheadphones 604). In some embodiments, headphones 604 amplify the ambientsound from physical environment 600 so that user 602 hears the ambientsound at a volume level above the volume level of ambient sound inphysical environment 600 (e.g., to assist users that are hard ofhearing).

FIG. 6D illustrates a transition from FIG. 6C. In particular, FIG. 6Dindicates that person 616 has stopped speaking and left physicalenvironment 600. As a result, the ambient sound present in physicalenvironment 600, as shown by ambient sound waveform 608, and theattenuated ambient sound, as shown by attenuated ambient sound waveform610, return to the same volume levels as shown in FIG. 6A. Accordingly,the ambient sound present in physical environment 600 again satisfiesthe predefined audio output criteria. In some embodiments, the ambientsound present in physical environment 600 in FIG. 6D is considered to nolonger satisfy the second predefined audio output criteria describedwith reference to FIGS. 6B-6C. In response, headphones 604 resumeplayback of the audio content from the device at the same volume levelas in FIG. 6A, as shown by device content waveform 606 in FIG. 6D.Headphones 604 also resume active noise control, and accordingly resumegenerating a noise-cancelling signal, as shown by antiphase waveform612, to cancel the attenuated ambient sound. As a result of the resumedactive noise control, no ambient sound is perceived by user 602, asindicated by perceived ambient sound waveform 614.

FIG. 6E illustrates an example scenario in which user 602 is notlistening to audio content from a device (e.g., when using headphones604 only for noise reduction and not for listening to audio content).Ambient sound present in physical environment 600 is shown by ambientsound waveform 608, and the attenuated ambient sound inside (e.g., theearcups of) headphones 604 (e.g., the ambient sound that is not blockedby headphones 604 as physical barriers over user 602's ears) is shown byattenuated ambient sound waveform 610. In FIG. 6E, because user 602 isnot listening to audio content from the device, as shown by devicecontent waveform 606, the balance between device audio content andambient sound from physical environment 600 is biased more toward theambient sound and less toward device audio content than in FIG. 6A, forexample. Thus, instead of generating an antiphase signal that completelycancels out the attenuated ambient sound from physical environment 600(as shown in FIG. 6A, for example), headphones 604 in FIG. 6E generatean antiphase signal, shown by antiphase waveform 612, that onlypartially cancels out attenuated ambient sound waveform 610. As aresult, user 602 perceives some, but not all, of the attenuated ambientsound, as shown by perceived ambient sound waveform 614 (e.g., user 602perceives some ambient sound, but at a volume level that is below thevolume level of even the attenuated ambient sound).

FIGS. 6F-6J illustrate initiating a temporary audio output state usingan input gesture via headphones 604. In FIG. 6F, as in FIG. 6A, user 602is seated and wearing headphones 604 in physical environment 600. User602 is listening to device audio content, as shown by device contentwaveform 606. Ambient sound present in physical environment 600 is shownby ambient sound waveform 608, and the attenuated ambient sound insideheadphones 604 is shown by attenuated ambient sound waveform 610.Headphones 604 also provide a noise-cancelling signal, shown byantiphase waveform 612, to cancel attenuated ambient sound waveform 610.As a result of the active noise control, no ambient sound is perceivedby user 602, as indicated by perceived ambient sound waveform 614. Inaddition, FIG. 6F illustrates user 602 providing input 618 (e.g., a tapgesture) via headphones 604 at time t=0, as indicated by timer 620. Asdescribed in more detail herein with reference to FIGS. 6G-6J, input 618initiates a temporary audio output state that lasts until time t=T_(th)(e.g., for a duration TO, as indicated by timer 620.

FIG. 6G illustrates a transition from FIG. 6F. In FIG. 6G, in responseto detecting input 618 via headphones 604, headphones 604 transition toa temporary audio output state. In the temporary audio output stateillustrated in FIG. 6G, the volume level of the device audio content islowered, as shown by device content waveform 606, and active noisecontrol is disabled, as shown by antiphase waveform 612. In addition,headphones 604 begin to actively pass through ambient sound fromphysical environment 600, so that user 602 hears ambient sound at avolume level that is above the attenuated ambient sound level resultingfrom passive attenuation by headphones 604 and that is approximatelyequal to (e.g., within 10 percent, 15 percent or 20 percent of) theactual volume level of ambient sound in physical environment 600, asshown by perceived ambient sound waveform 614.

FIG. 6H illustrates a transition from FIG. 6G. In FIG. 6H, anotherperson 622 has approached user 602 and is speaking to user 602. Theincrease in ambient sound in physical environment 600 due to person 622speaking is shown by ambient sound waveform 608 and by attenuatedambient sound waveform 610. Because headphones 604 are in the temporaryaudio output state (e.g., because t<T_(th), as indicated by timer 620),headphones 604 continue to output audio content from the device at areduced volume level, as shown by device content waveform 606.Headphones 604 also continue to pass through ambient sound from physicalenvironment 600 without active noise control, as shown by antiphasewaveform 612. As a result, user 602 hears person 622 speaking and otherambient sound at a volume level that is above the attenuated ambientsound level resulting from passive attenuation by headphones 604 andthat is approximately equal to (e.g., within 10 percent, 15 percent or20 percent of) the actual volume level of speech and other ambient soundin physical environment 600, as shown by perceived ambient soundwaveform 614.

FIG. 6I illustrates a transition from FIG. 6H. In particular, FIG. 6Iindicates that person 622 has stopped speaking and left physicalenvironment 600. As a result, the ambient sound present in physicalenvironment 600, as shown by ambient sound waveform 608, and theattenuated ambient sound, as shown by attenuated ambient sound waveform610, return to the same volume levels as shown in FIG. 6G. In addition,because headphones 604 are still in the temporary audio output state(e.g., because t<T_(th), as indicated by timer 620), headphones 604continue to output audio content from the device at a reduced volumelevel, as shown by device content waveform 606. Headphones 604 alsocontinue to pass through ambient sound from physical environment 600without active noise control, as shown by antiphase waveform 612. As aresult, user 602 hears ambient sound at a volume level that is above theattenuated ambient sound level resulting from passive attenuation byheadphones 604 and that is approximately equal to (e.g., within 10percent, 15 percent or 20 percent of) the actual volume level of ambientsound in physical environment 600, as shown by perceived ambient soundwaveform 614.

FIG. 6J illustrates a transition from FIG. 6I. In FIG. 6J, the temporaryaudio output state initiated by input 618 (FIG. 6F) has expired, asindicated by timer 620 showing t=T_(th). As a result, headphones 604resume providing audio outputs at the same volume levels as before thetemporary audio output state was initiated. In particular, headphones604 resume playback of device audio content at the same volume level asin FIG. 6F, as shown by device content waveform 606 in FIG. 6J.Headphones 604 also resume active noise control, and accordingly resumegenerating a noise-cancelling signal, as shown by antiphase waveform612, to cancel the attenuated ambient sound. As a result of the resumedactive noise control, no ambient sound is perceived by user 602, asindicated by perceived ambient sound waveform 614. In an example furthertransition from FIG. 6J, in which another person approaches and beginsspeaking to user 602 after the temporary audio output state has expired(and before another temporary audio output state is initiated),headphones 604 respond to the change in the ambient sound in physicalenvironment 600 as described herein with reference to FIGS. 6A-6C.

FIGS. 7A-7B are flow diagrams illustrating method 700 of adaptivelychanging simulated spatial locations of audio outputs in response tochanges in user pose in accordance with some embodiments. Method 700 isperformed at an electronic device (e.g., portable multifunction device100, FIG. 1A, device 300, FIG. 3A, or wearable audio output device 301,FIG. 3B) that includes or is in communication with one or more posesensors for detecting a pose (e.g., orientation and/or position) of auser of the electronic device relative to a first physical environment,and that includes or is in communication with one or more audio outputdevices (e.g., speaker 111, FIG. 1A, or speaker(s) 306 of wearable audiooutput device 301, FIG. 3B). In some embodiments, the pose sensorsinclude one or more cameras (e.g., optical sensor(s) 164, FIG. 1A orsensor(s) 359, FIG. 3A), gyroscopes, inertial measurement units, orother sensors (e.g., accelerometer(s) 168, FIG. 1A or sensor(s) 359,FIG. 3A) that enable the electronic device to detect changes in anorientation (also called attitude) and/or position of the user. In someembodiments, the electronic device detects changes in the pose of theuser by detecting changes in the orientation and/or position of theelectronic device or of component(s) of the electronic device (e.g., adisplay generation component, such as touch-sensitive display system 112in FIG. 1A, display 340 in FIG. 3A, a projector, a heads-up display, ahead-mounted display, etc.). In some embodiments, the electronic devicedetects changes in the pose of the user by detecting changes in theorientation and/or position of component(s) with which the electronicdevice is in communication and that are held or worn by the user (e.g.,using pose sensor(s) 304 of wearable audio output device 301, FIG. 3B),relative to a physical environment in which the electronic device islocated. Some operations in method 700 are, optionally, combined and/orthe order of some operations is, optionally, changed.

As described below, method 700 provides audio outputs in an intuitivemanner by varying output properties of the audio output, such as spatiallocation, volume, content, etc. of the audio output, in response tochanges in a user's pose (e.g., position and/or orientation). It isnoted that spatial location, sometimes called simulated spatiallocation, is a perceptual property of audio outputs. Spatial locationcan be controlled or varied using well known audio synthesis techniques,so as to make audio outputs be perceived as coming from a particularspatial location in three-dimensional space that is different from thephysical location of the speakers that produce the audio outputs.Generally, at least two speakers are required to vary the spatiallocation of an audio output. Varying output properties of audio output,such as by moving the simulated spatial location of the audio output, inresponse to changes in user pose allows for the manner in which audiooutput is provided to be dynamically adjusted to be better suited forthe current activity of the user, without requiring additional inputfrom the user. Providing an adaptive and more intuitive user experiencewhile reducing the number of inputs needed to achieve such an experienceenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by helping the user to achieve anintended outcome and reducing user mistakes when operating/interactingwith the device), which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

While a first pose of the user meets first presentation criteria (702),the device provides (704) audio content at a first simulated spatiallocation relative to the user. In some embodiments, the audio isprovided via one or more audio output devices (e.g., wearable audiooutput device 301, FIG. 3B) that are in communication with (and, in someembodiments, separate from) the electronic device. In some embodiments,the pose of the user corresponds to and indicates that the user is in aparticular position (e.g., lying down, reclining/leaning back, sitting,or standing). In some embodiments, the pose of the user meets the firstpresentation criteria when the first pose of the user is in a predefinedrange or set of poses (e.g., corresponding to a particular position ofthe user, such as when the user is reclined to within a predefinedangular range with respect to horizontal). For example, as describedherein with reference to FIG. 5A, FIG. 5A illustrates audio contentbeing provided in simulation spatial region 506 while user 502 issitting upright. In some embodiments, the pose of the user meets thefirst presentation criteria when the user is in a particular type ofspace (e.g., a private space). For example, as described herein withreference to FIG. 5G, FIG. 5G illustrates audio content being providedin simulated spatial region 506 while user 502 is in private space 512.

The device detects (706) a change in the pose of the user from the firstpose to a second pose. In some embodiments, the first pose of the useris (708) one of a lying down pose, a sitting pose, and a standing pose,and the second pose of the user is a different one of a lying down pose,a sitting pose, and a standing pose. In some embodiments, the userchanges pose when moving between standing, sitting, and lying downpositions. For example, as described herein with reference to FIGS.5A-5B, FIGS. 5A-5B illustrate a change in the pose of user 502 fromsitting (FIG. 5A) to standing (FIG. 5B). In another example, asdescribed herein with reference to FIGS. 5I-5K, FIGS. 5I-5K illustrate achange in the pose of user 502 from reclining (FIG. 5I) to sittingupright (FIGS. 5I-5K).

In some embodiments, the degree of immersion of the provided audio ishighest when a user is lying down, and lowest when a user is standing upor moving about the physical environment. In some circumstances, when auser is lying down, audio is provided at one or more simulated spatiallocations inside or surrounding the user's head, such that the audiosounds as though it were coming from or being played inside orsurrounding the user's head. In some other circumstances, when a userstands up, the audio is provided at a simulated spatial location abovethe user's head, such that the audio sounds as though it were comingfrom or being played above the user's head, and optionally at a lowervolume.

In some embodiments where a simulated three-dimensional environment thatincludes one or more virtual objects is displayed on, or using, adisplay generation component in conjunction with providing audio at arespective simulated spatial location, the degree of immersion of thedisplayed simulated three-dimensional environment is highest when a useris lying down, and lowest when a user is standing up or moving about thephysical environment. In some circumstances, when a user is lying down,the simulated three-dimensional environment includes more virtualobjects and fewer (or no) representations of physical objects in thephysical environment. In other circumstances, when a user sits up from alying down position, the electronic device ceases to display at least aportion of one or more virtual objects and/or displays representationsof additional portions of the physical environment (e.g., displayingadditional representations of physical objects in the physicalenvironment). In other circumstances, when a user stands up, theelectronic device ceases to display even more virtual objects orportions of virtual objects and/or displays representations of even moreof the physical environment.

Moving the simulated spatial location at which audio content is providedbased on changes in user pose between lying down, sitting, and/orstanding allows for the manner in which audio output is provided to bedynamically adjusted to be better suited for the current activity of theuser (e.g., inferred based on whether the user is lying down, sitting,or standing), without requiring additional input from the user.Providing an adaptive and more intuitive user experience while reducingthe number of inputs needed to achieve such an experience enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to achieve an intended outcome andreducing user mistakes when operating/interacting with the device),which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In some embodiments, detecting the change in the pose of the userincludes (710) detecting a change in the position of the user from thefirst physical environment to a second physical environment that isdifferent from the first physical environment (e.g., movement of theuser from one room in a building to a different room in a building; ormovement of the user from a first type of physical environment to asecond type of physical environment, such as between private and publicspaces, for example from a bedroom to a living room or vice versa). Forexample, as described herein with reference to FIGS. 5F-5G, FIGS. 5F-5Gillustrate a change in the pose of user 502 through movement of user 502from public space 510 to private space 512.

Moving the simulated spatial location at which audio content is providedbased on movement of the user between private and public spaces allowsfor the manner in which audio output is provided to be dynamicallyadjusted to be better suited for the current environment of the user(e.g., inferred based on whether the user is in a public space, in whichless audio immersion is typically wanted, or in a private space, inwhich greater audio immersion is typically wanted), without requiringadditional input from the user. Providing an adaptive and more intuitiveuser experience while reducing the number of inputs needed to achievesuch an experience enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user toachieve an intended outcome and reducing user mistakes whenoperating/interacting with the device), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In response to detecting the change in the pose of the user, and inaccordance with a determination that the second pose of the user doesnot meet the first presentation criteria (712), the device provides(714) audio content at a second simulated spatial location relative tothe user that is different from the first simulated spatial location(e.g., via the one or more audio output devices). For example, asdescribed herein with reference to FIGS. 5A-5B, 5F-5G, and 5I-5K, thesimulated spatial location of the audio output (e.g., music) is moved inresponse to detected changes in user pose. In some embodiments, theprovided audio corresponds to audio content from the electronic device.In some embodiments, the audio content is stored on the electronicdevice. In some embodiments, the audio content is obtained by theelectronic device from an external source (e.g., the Internet, a contentstreaming source, or the like). In some embodiments, the audio contentfrom the electronic device is distinct from sound in the physicalenvironment (e.g., sound detected via a microphone directed toward thephysical environment).

In some embodiments, the second simulated spatial location is (716)above the user. In some embodiments, providing the audio at the secondsimulated spatial location includes providing the audio so that theaudio sounds as though it were coming from above the user. For example,as described herein with reference to FIGS. 5B, 5F, and 5K, audio output(e.g., music) is provided at simulated spatial location 508 that isabove user 502's head. Moving the simulated spatial location at whichaudio content is provided to a simulated spatial location above the userin response to a change in user pose reduces the level of immersion ofthe provided audio content so that the user can better interact with thesurrounding physical environment, without requiring additional inputfrom the user. Providing an adaptive and more intuitive user experiencewhile reducing the number of inputs needed to achieve such an experienceenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by helping the user to achieve anintended outcome and reducing user mistakes when operating/interactingwith the device), which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In some embodiments, the device detects (718) a change in the pose ofthe user from the second pose to a third pose, and, in response todetecting the change in the pose of the user from the second pose to thethird pose, and in accordance with a determination that the third poseof the user meets the first presentation criteria, the device providesaudio content at the first simulated spatial location relative to theuser. For example, as described herein with reference to FIGS. 5C-5D,FIGS. 5C-5D illustrate user resuming a seated pose (FIG. 5D) afterpreviously assuming a standing pose (e.g., FIGS. 5B-5C illustrating user502 standing up from being seated as illustrated in FIG. 5A). Inresponse, and because user 502's seated pose in FIG. 5D also meets thepresentation criteria (e.g., that were previously met by user 502'sseated pose in FIG. 5A), the simulated spatial location of the audiooutput being provided is moved back to simulated spatial region 506. Insome embodiments, the third pose is the same as the first pose. In someembodiments, in conjunction with providing the audio content at thefirst simulated spatial location relative to the user, the deviceprovides (or, in some embodiments, resumes providing) video content.

Moving the simulated spatial location at which audio content is providedto a previous location in response to the user resuming a correspondingprevious pose allows for the manner in which audio output is provided tobe dynamically adjusted to be better suited for the current environmentof the user, without requiring additional input from the user. Providingan adaptive and more intuitive user experience while reducing the numberof inputs needed to achieve such an experience enhances the operabilityof the device and makes the user-device interface more efficient (e.g.,by helping the user to achieve an intended outcome and reducing usermistakes when operating/interacting with the device), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In some embodiments, the device detects (720), via one or moremicrophones (e.g., microphone 113, FIG. 1A or microphone(s) 302, FIG.3B), sound in the physical environment. For example, as described hereinwith reference to FIG. 6B, FIG. 6B illustrates detection of an increasein sound (e.g., due to person 616 beginning to speak) in physicalenvironment 600. In some embodiments, providing the audio content at thesecond simulated spatial location includes reducing an output level ofthe audio content relative to the detected sound from the physicalenvironment. In some embodiments, reducing the output level of theprovided audio content relative to the detected sound from the physicalenvironment includes reducing a volume of the provided audio content.For example, as described herein with reference to FIG. 6C, FIG. 6Cillustrates reduction of the volume of device audio content. In someembodiments, the audio content is paused (e.g., after at least a portionof the audio content is provided at the second simulation spatiallocation). In some embodiments, reducing the output level of theprovided audio content relative to the detected sound from the physicalenvironment includes reducing an amount of noise-cancellation (e.g.,reducing or ceasing to provide audio configured to cancel ambient sound,sometimes called “antiphase” audio). For example, as described hereinwith reference to FIG. 6C, FIG. 6C illustrates that active noise controlhas been disabled. In some embodiments, reducing the output level of theprovided audio content relative to the detected sound from the physicalenvironment includes providing (e.g., beginning to provide, orincreasing an amount of) audio corresponding to the detected sound fromthe physical environment (e.g., by actively providing at least a portionof the detected sound from the physical environment). For example, asdescribed herein with reference to FIG. 6C, FIG. 6C illustrates activepassthrough of ambient sound from physical environment 600.

In some embodiments, the output level of the provided audio content isreduced relative to the detected sound from the physical environmentgradually (e.g., the audio content gradually fades away). In someembodiments, the simulated spatial location of the provided audiocontent is changed from the first simulated spatial location to thesecond spatial location gradually (e.g., the audio content sounds as ifit is moving gradually to a different simulated spatial location). Insome embodiments, the gradual reduction in the output level of theprovided audio content relative to the detected sound from the physicalenvironment is performed in concert with gradually changing thesimulated spatial location of the provided audio content from the firstsimulated spatial location to the second simulated spatial location(e.g., the audio content sounds as if it is moving gradually to adifferent simulated spatial location while also fading away).

Reducing an output level (such as the volume) of provided audio contentrelative to sound detected in the surrounding physical environmentreduces the level of immersion of the provided audio content so that theuser can better interact with the surrounding physical environment,without requiring additional input from the user. Providing an adaptiveand more intuitive user experience while reducing the number of inputsneeded to achieve such an experience enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byhelping the user to achieve an intended outcome and reducing usermistakes when operating/interacting with the device), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In some embodiments, the device determines (722) an event associatedwith the change in the pose of the user from the first pose to thesecond pose (e.g., the device determines an event that occurredimmediately or within a predefined time period prior to detecting thechange in the user's pose, that can be inferred to have caused the userto change pose), and, in response to detecting the change in the pose ofthe user, and in accordance with the determination that the second poseof the user does not meet the first presentation criteria, the deviceperforms an operation to assist the user to respond to the event. Insome examples, the event includes another person entering the room andspeaking (e.g., to the user), and the operation performed to assist theuser to respond to the event includes ceasing to provide at least aportion of media content being provided (e.g., lowering the volume of orpausing audio being played, increasing transparency of or pausing and/orceasing to display a video being presented, etc.). In other examples,the event includes a notification, such as an alert for an upcomingcalendar event, and the operation performed to assist the user torespond to the event includes operations such as displaying details forthe upcoming calendar event, launching a maps application withnavigation directions to the location of the upcoming calendar event,and/or providing reminders of one or more physical objects to retrievefor the upcoming calendar event (e.g., keys, wallet, coat, and/or itemsneeded at the upcoming calendar event). For example, as described hereinwith reference to FIGS. 5I-5J and 5L, FIGS. 5I-5J illustrate user 502changing pose by sitting up from a reclined pose in response tonotification 520, and FIG. 5L illustrates display of beacons 524-1 and524-2 to help user 502 respond to the event, notification 520, thatprompted user 502 to change pose.

Assisting the user to respond to an event that caused the user to changepose, such as by providing reminders of physical objects to take alongto a meeting, or reducing a level of immersion of provided audio and/orvideo content, allows for the manner in which audio and/or visualoutputs are provided to be dynamically adjusted to be better suited forthe current activity of the user, and helps the user to better interactwith the surrounding physical environment, without requiring additionalinput from the user. Providing an adaptive and more intuitive userexperience while reducing the number of inputs needed to achieve such anexperience enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user toachieve an intended outcome and reducing user mistakes whenoperating/interacting with the device), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In some embodiments, the electronic device includes (or is incommunication with) a display generation component (e.g., a display suchas display 340 in FIG. 3A, a touchscreen such as touch-sensitive displaysystem 112 in FIG. 1A, a projector, a heads-up display, a head-mounteddisplay, etc.). In some embodiments, while the first pose of the usermeets the first presentation criteria (724), the device displays, viathe display generation component, video content corresponding to theaudio content, and, in response to detecting the change in the pose ofthe user, and in accordance with a determination that the second pose ofthe user does not meet the first presentation criteria, the deviceceases to display at least a portion of the video content (e.g., bypausing playback of the video content). For example, as described hereinwith reference to FIGS. 5H-5L, FIGS. 5H-5I illustrates playback of video516 while user 502 is reclined, and, in response to user 502 sitting upin FIG. 5J, playback of video 516 is paused, as shown in FIGS. 5J-5L. Insome embodiments, the video content corresponding to the audio contentis stored on the electronic device, optionally in conjunction with theaudio content. In some embodiments, the video content is obtained by theelectronic device from an external source (e.g., the Internet, a contentstreaming source, or the like), optionally in conjunction with the audiocontent. In some embodiments, the video content is distinct from a view(e.g., a representation) of the physical environment (e.g., a live viewof a camera directed toward the physical environment). For example,video 516 (FIG. 5K) is distinct from live view 522 (FIG. 5K).

Where video content is provided in conjunction with providingcorresponding audio content, ceasing to display at least a portion ofthe video content in response to changes in user pose reduces the levelof immersion of the provided video content to help the user readjust toand better interact with the surrounding physical environment, withoutrequiring additional input from the user. Providing an adaptive and moreintuitive user experience while reducing the number of inputs needed toachieve such an experience enhances the operability of the device andmakes the user-device interface more efficient (e.g., by helping theuser to achieve an intended outcome and reducing user mistakes whenoperating/interacting with the device), which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In some embodiments where the electronic device includes a displaygeneration component, while the first pose of the user meets the firstpresentation criteria (726), the device displays, via the displaygeneration component, a simulated three-dimensional environment thatincludes one or more virtual objects, and, in response to detecting thechange in the pose of the user, and in accordance with a determinationthat the second pose of the user does not meet the first presentationcriteria, the device ceases to display at least a portion of one or morevirtual objects in the simulated three-dimensional environment. Forexample, as described herein with reference to FIGS. 5H-5L, FIGS. 5H-5Iillustrates playback of video 516 (e.g., as a full-screen virtual objectin a simulated three-dimensional environment) while user 502 isreclined, and, in response to user 502 sitting up in FIG. 5J, playbackof video 516 is paused, as shown in FIGS. 5J-5L, and, optionally, thetransparency of video 516 is increased, as shown in FIG. 5L.

Where a simulated three-dimensional environment (e.g., an augmentedand/or virtual reality environment) is provided in conjunction withproviding corresponding audio content, ceasing to display at least aportion of the video content in response to changes in user pose reducesthe level of immersion of the simulated three-dimensional environment tohelp the user readjust to and better interact with the surroundingphysical environment, without requiring additional input from the user.Providing an adaptive and more intuitive user experience while reducingthe number of inputs needed to achieve such an experience enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to achieve an intended outcome andreducing user mistakes when operating/interacting with the device),which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In some embodiments, in response to detecting the change in the pose ofthe user (728), and in accordance with the determination that the secondpose of the user does not meet the first presentation criteria, thedevice displays, via the display generation component, a representationof at least a portion of the first physical environment. For example, asdescribed herein with reference to FIGS. 5J-5L, in response to user 502sitting up in FIG. 5J, live view 522 is at least partially displayed, asshown in FIGS. 5K-5L. In some embodiments, the electronic devicedisplays representations (e.g., live views) of physical objects in thefirst physical environment (e.g., plant 518, wallet 526, and keys 528,FIGS. 5K-5L). In some embodiments, the electronic device displays ananimated transition from the simulated three-dimensional environment tothe representation of at least the portion of the physical environment.In some embodiments, the electronic device increases an apparenttransparency of video content and/or other virtual objects displayed inthe simulated three-dimensional environment in conjunction withdisplaying the representation of at least the portion of the firstphysical environment (e.g., to create the illusion that the physicalenvironment is becoming visible “through” the video content and/or othervirtual objects). For example, FIG. 5L illustrates that the transparencyof video 516 (which in some embodiments is considered to be a virtualobject) has been increased so that live view 522 of physical environment500, including physical objects plant 518, wallet 526, and keys 528, areat least partially visible “through” video 516.

Displaying a representation (e.g., a live view from a camera) of atleast a portion of a surrounding physical environment in response tochanges in user pose reduces the level of immersion of a displayedsimulated three-dimensional environment or other displayed userinterface to help the user readjust to and better interact with thesurrounding physical environment, without requiring additional inputfrom the user. Providing an adaptive and more intuitive user experiencewhile reducing the number of inputs needed to achieve such an experienceenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by helping the user to achieve anintended outcome and reducing user mistakes when operating/interactingwith the device), which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In some embodiments, in response to detecting the change in the pose ofthe user (730), and in accordance with the determination that the secondpose of the user does not meet the first presentation criteria, thedevice changes respective positions of one or more virtual objects inthe simulated three-dimensional environment. In some embodiments, theone or more virtual objects are moved out of a particular area in thesimulated three-dimensional environment (e.g., so as to clear an areafor displaying/viewing a live view of the physical environment). In someembodiments, where video content is displayed in the simulatedthree-dimensional environment, the video content is displayed at a firstlocation (e.g., a location in a direction that the user is facing) whilethe first pose of the user meets the first presentation criteria. Insome embodiments, in response to detecting the change in the pose of theuser to a second pose that does not meet the first presentationcriteria, the video content is moved to a second location (e.g., alocation above or to the side of an area in the direction that the useris facing). For example, as described herein with reference to FIGS.5J-5K, in response to user 502 sitting up in FIG. 5J, video 516 isscaled down and moved to a corner of the displayed simulatedthree-dimensional environment, as shown in FIG. 5K.

Moving the positions of one or more virtual objects displayed in asimulated three-dimensional environment in response to changes in userpose (e.g., so as to clear an area for displaying a live view of thesurrounding physical environment) reduces the level of immersion of thedisplayed simulated three-dimensional environment to help the userreadjust to and better interact with the surrounding physicalenvironment, without requiring additional input from the user. Providingan adaptive and more intuitive user experience while reducing the numberof inputs needed to achieve such an experience enhances the operabilityof the device and makes the user-device interface more efficient (e.g.,by helping the user to achieve an intended outcome and reducing usermistakes when operating/interacting with the device), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

It should be understood that the particular order in which theoperations in FIGS. 7A-7B have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,method 800) are also applicable in an analogous manner to method 700described above with respect to FIGS. 7A-7B. For example, the audiooutputs, simulated spatial locations, and physical environmentsdescribed above with reference to method 700 optionally have one or moreof the characteristics of the poses, presentation criteria, audiocontent, simulated spatial locations, and physical environmentsdescribed herein with reference to other methods described herein (e.g.,method 800). For brevity, these details are not repeated here.

FIGS. 8A-8B are flow diagrams illustrating method 800 of adaptivelychanging audio output levels in response to changes in the audioproperties of a surrounding physical environment in accordance with someembodiments. Method 800 is performed at one or more wearable audiooutput devices (e.g., wearable audio output device 301, FIG. 3B) thatare in a respective physical environment and that are in communication(e.g., via a wireless connection, via a wired connection, or integrated)with an electronic device (e.g., portable multifunction device 100, FIG.1A or device 300, FIG. 3). Some operations in method 800 are,optionally, combined and/or the order of some operations is, optionally,changed.

As described below, method 800 provides audio outputs in an intuitivemanner by adjusting the levels of different audio components in an audiooutput in response to changes in the audio properties of the surroundingphysical environment. In some examples, the provided audio output isautomatically adjusted to allow the user to hear more ambient sound whenspeech (or an increase in speech) is detected in the surroundingphysical environment. Varying the levels of different audio components,and the balance between the different audio components, in an audiooutput allows for the manner in which audio output is provided to bedynamically adjusted to be better suited to the current state of thesurrounding physical environment, without requiring additional inputfrom the user. Providing an adaptive and more intuitive user experiencewhile reducing the number of inputs needed to achieve such an experienceenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by helping the user to achieve anintended outcome and reducing user mistakes when operating/interactingwith the device), which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

While one or more audio properties of the respective physicalenvironment satisfy first audio criteria (802), the wearable audiooutput device(s) provide audio output corresponding to the first audiocriteria. In some embodiments, the one or more audio properties of therespective physical environment satisfy the first audio criteria whenthe respective physical environment includes a respective type of sound,such as white noise, or when an energy density of sound (e.g.,corresponding to a respective range of frequencies, and/or within arespective moving window of time) in the respective physical environmentmeets a predefined threshold. The provided audio output includes: audiocorresponding to audio content from the electronic device at a firstdevice-content audio level; and audio corresponding to ambient soundfrom the respective physical environment at a first ambient-sound audiolevel. In some embodiments, the ambient-sound audio level (e.g., of theaudio corresponding to ambient sound from the respective physicalenvironment) is zero (e.g., no ambient sound is being actively passedthrough by the one or more wearable audio output devices, although auser wearing the one or more wearable audio output devices may perceivesome ambient sound due to imperfect passive attenuation by the wearableaudio output devices). For example, as described herein with referenceto FIG. 6A, while the ambient sound present in physical environment 600satisfies predefined audio output criteria, headphones 604 providedevice audio content at the level shown by device content waveform 606in FIG. 6A, and do not actively pass through ambient sound from physicalenvironment 600. In another example, as described herein with referenceto FIG. 6C, while the ambient sound present in physical environment 600satisfies different (e.g., second) predefined audio output criteria,headphones 604 provide device audio content at the level shown by devicecontent waveform 606 in FIG. 6C, and actively pass through ambient soundfrom physical environment (e.g., at a non-zero level), as described withreference to perceived ambient sound waveform 614.

The wearable audio output device(s) detect (804) a change in the one ormore audio properties of the respective physical environment. In someembodiments, detecting a change in the one or more audio properties ofthe respective physical environment includes detecting a change inspeech relative to the ambient sound (e.g., detecting an increase inspeech corresponding to a person speaking or beginning to speak, ordetecting a decrease in speech corresponding to a person finishingspeaking). For example, as described herein with reference to FIG. 6B,FIG. 6B illustrates an increase in speech by person 616 in physicalenvironment 600. In another example, as described herein with referenceto FIG. 6D, FIG. 6D illustrates a decrease in speech in physicalenvironment 600.

In response to detecting the change in the one or more audio propertiesof the respective physical environment (806), the wearable audio outputdevice(s) provide audio corresponding to ambient sound from therespective physical environment at a second ambient-sound audio levelthat is different from the first ambient-sound audio level. For example,as described herein with reference to FIGS. 6B-6C, in response todetecting the increase in speech by person 616, headphones 604 change(e.g., increase) the level at which ambient sound from physicalenvironment 600 is provided to user 602, in part by beginning to passthrough ambient sound from physical environment 600, as described withreference to perceived ambient sound waveform 614 in FIG. 6C. In anotherexample, as described herein with reference to FIG. 6D, in response todetecting the decrease in speech by person 616, headphones 604 change(e.g., decrease) the level at which ambient sound from physicalenvironment 600 is provided to user 602, in part by ceasing to passthrough ambient sound from physical environment 600, as described withreference to perceived ambient sound waveform 614 in FIG. 6D.

In some embodiments, the detecting and adjusting are performed usinghardware circuitry and/or software modules on the one or more wearableaudio output devices for better responsiveness to changes in thephysical environment. In some examples, detecting the change in the oneor more audio properties of the respective physical environment includesdetecting an increase in speech in the physical environment, and thesecond ambient-sound audio level is greater than the first ambient-soundaudio level (e.g., the amount of ambient sound passed through from thephysical environment is increased in response to detecting an increasein speech in the physical environment, as described herein withreference to FIGS. 6B-6C).

In some embodiments, in response to detecting the change in the one ormore audio properties of the respective physical environment (808), thewearable audio output device(s) provide audio corresponding to audiocontent from the electronic device at a second device-content audiolevel that is different from the first device-content audio level. Insome embodiments where detecting the change in the one or more audioproperties of the physical environment includes detecting an increase in(e.g., beginning of) speech, the device-content audio level (e.g., ofthe audio corresponding to audio content from the electronic device) isdecreased (e.g., to a lower device-content audio level above zero) or insome cases paused (e.g., the device-content audio level is decreased tozero). For example, as described herein with reference to FIGS. 6B-6C,in response to detecting the increase in speech by person 616,headphones 604 change (e.g., decrease) the level at which audio contentfrom the device is provided, as described with reference to devicecontent waveform 606 in FIG. 6C. In some embodiments where detecting thechange in the one or more audio properties of the physical environmentincludes detecting a decrease in (e.g., end of) speech, thedevice-content audio level (e.g., of the audio corresponding to audiocontent from the electronic device) is increased (e.g., from a lowerdevice-content audio level above zero to a higher device-content audiolevel) or in some cases resumed (e.g., the device-content audio level isincreased from zero). For example, as described herein with reference toFIG. 6D, in response to detecting the decrease in speech by person 616,headphones 604 change (e.g., increase) the level at which audio contentfrom the electronic device is provided, as described with reference todevice content waveform 606 in FIG. 6D.

Changing the level of provided audio content from a device in responseto changes in the audio properties of the surrounding physicalenvironment balances the user's ability to hear provided device audiocontent against the user's ability to interact with the surroundingphysical environment, without requiring additional input from the user.Providing an adaptive and more intuitive user experience while reducingthe number of inputs needed to achieve such an experience enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to achieve an intended outcome andreducing user mistakes when operating/interacting with the device),which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In some embodiments, the one or more wearable audio output devicesinclude one or more pose sensors (e.g., pose sensor(s) 304, FIG. 3B) fordetecting a pose of the one or more wearable audio output devices. Insome embodiments, the one or more pose sensors include one or moregyroscopes, inertial measurement units, or other sensors that enable theone or more wearable audio output devices to detect changes in anorientation and/or position of the one or more wearable audio outputdevices or of component(s) of the one or more wearable audio outputdevices (e.g., when worn by a user), relative to a physical environmentin which the one or more wearable audio output devices are located. Insome embodiments, the wearable audio output device(s) detect a change inthe pose of the wearable audio output device(s), and the audiocorresponding to ambient sound from the respective physical environmentis provided at the second ambient-sound audio level further in responseto detecting the change in the pose of the one or more wearable audiooutput devices (e.g., in addition to being provided in response todetecting the change in the one or more audio properties of therespective physical environment). For example, the audio outputsdescribed herein with reference to FIGS. 5A-5L can include differenttypes of audio (e.g., device audio content, ambient audio, or antiphaseaudio), as described herein with reference to FIGS. 6A-6J, and thevolume levels of the different types of audio can be changed in responseto changes in user pose (e.g., as described herein with reference toFIGS. 5A-5B, 5F-5G, and 5I-5K) instead of or in addition to beingchanged in response to changes in the audio properties of thesurrounding physical environment).

Adjusting the levels of different audio components in an audio output inresponse to changes in user pose (as well as changes in the audioproperties of the surrounding physical environment) allows for themanner in which audio output is provided to be dynamically adjusted tobe better suited for the current activity of the user (in addition tobeing better suited to the current state of the surrounding physicalenvironment), without requiring additional input from the user.Providing an adaptive and more intuitive user experience while reducingthe number of inputs needed to achieve such an experience enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to achieve an intended outcome andreducing user mistakes when operating/interacting with the device),which, additionally, reduces power usage and improves battery life ofthe device by enabling the user to use the device more quickly andefficiently.

In some embodiments, while the one or more audio properties of therespective physical environment satisfy the first audio criteria (812),the audio output further includes audio configured to cancel at least aportion of ambient sound from the respective physical environment at afirst audio-cancelling audio level. In some embodiments, the audio atthe first audio-cancelling audio level is greater than zero, andconfigured to cancel at least a portion of ambient sound from therespective physical environment that otherwise would be perceived by theuser due to imperfect passive attenuation by the wearable audio outputdevices, even if the ambient-sound audio level for audio activelyprovided by the one or more wearable audio output devices is zero. Forexample, as described herein with reference to FIG. 6A, antiphasewaveform 612 is provided and configured to cancel attenuated ambientsound waveform 610, even when headphones 604 are not actively passingambient sound through from physical environment 600. In someembodiments, in response to detecting the change in the one or moreaudio properties of the respective physical environment, the wearableaudio output device(s) provide audio configured to cancel at least aportion of ambient sound from the respective physical environment at asecond audio-cancelling audio level that is different from the firstaudio-cancelling audio level.

In some embodiments where detecting the change in the one or more audioproperties of the physical environment includes detecting an increase in(e.g., beginning of) speech, the audio-cancelling audio level (e.g., ofthe audio configured to cancel at least a portion of ambient sound fromthe respective physical environment) is decreased (e.g., to a loweraudio-cancelling audio level above zero) or in some cases stopped (e.g.,the audio-cancelling audio level is decreased to zero). For example, asdescribed herein with reference to FIGS. 6B-6C, in response to detectingthe increase in speech by person 616, headphones 604 decrease (e.g., tozero) the level of antiphase audio that is provided, as described withreference to antiphase waveform 612. In some embodiments where detectingthe change in the one or more audio properties of the physicalenvironment includes detecting a decrease in (e.g., end of) speech, theaudio-cancelling audio level (e.g., of the audio configured to cancel atleast a portion of ambient sound from the respective physicalenvironment) is increased (e.g., from a lower audio-cancelling audiolevel above zero to a higher audio-cancelling audio level) or in somecases resumed (e.g., the audio-cancelling audio level is increased fromzero). For example, as described herein with reference to FIG. 6D, inresponse to detecting the decrease in speech by person 616, headphones604 increase the level of antiphase audio that is provided, as describedwith reference to antiphase waveform 612.

Changing the level of noise-cancellation in response to changes in theaudio properties of the surrounding physical environment balances theamount of noise reduction against the user's ability to interact withthe surrounding physical environment, without requiring additional inputfrom the user. Providing an adaptive and more intuitive user experiencewhile reducing the number of inputs needed to achieve such an experienceenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by helping the user to achieve anintended outcome and reducing user mistakes when operating/interactingwith the device), which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In some embodiments, detecting the change in the one or more audioproperties of the respective physical environment includes (814)detecting an increase in speech in the physical environment, and, inresponse to detecting the change in the one or more audio properties ofthe respective physical environment, the second audio-cancelling audiolevel is zero. Stated another way, in some embodiments, in response todetecting an increase in speech in the physical environment, the one ormore wearable audio output devices stop providing the audio configuredto cancel at least a portion of ambient sound. For example, as describedherein with reference to FIGS. 6B-6C, in response to detecting theincrease in speech by person 616, headphones 604 decrease the level ofantiphase audio that is provided to zero, as described with reference toantiphase waveform 612. Turning off noise-cancellation (e.g., reducingthe level of noise-cancellation to zero) in response to detecting anincrease in speech in the surrounding physical environment helps theuser to better interact with the surrounding physical environment,without requiring additional input from the user. Providing an adaptiveand more intuitive user experience while reducing the number of inputsneeded to achieve such an experience enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byhelping the user to achieve an intended outcome and reducing usermistakes when operating/interacting with the device), which,additionally, reduces power usage and improves battery life of thedevice by enabling the user to use the device more quickly andefficiently.

In some embodiments, the audio configured to cancel at least a portionof ambient sound from the respective physical environment is (816) basedon an amount by which ambient sound from the respective physicalenvironment is reduced by the one or more wearable audio output deviceswhen worn by a user. In some embodiments, the one or more wearable audiooutput devices determine the amount by which ambient sound from therespective physical environment is reduced by the one or more wearableaudio output devices when worn by a user (sometimes called “passiveattenuation”) and use the determined amount in generating the audioconfigured to cancel ambient sound (sometimes called “antiphase” audio).In some embodiments, the one or more wearable audio output devicesinclude one or more sensors for determining the passive attenuation ofthe one or more wearable audio output devices. For example, as describedherein with reference to FIG. 3C, where the one or more wearable audiooutput devices are over-ear headphones, the headphones include one ormore first sensors configured to measure ambient sound in the respectivephysical environment and one or more second sensors configured tomeasure ambient sound perceivable by the user when wearing the one ormore wearable audio output devices. In some embodiments, the one or morefirst sensors include one or more microphones (e.g., microphone(s)302-1, FIG. 3C) measuring ambient sound outside the headphone earcup(s)(e.g., represented by waveform 322, FIG. 3C). In some embodiments, theone or more second sensors include one or more microphones (e.g.,microphone(s) 302-2, FIG. 3C) measuring ambient sound inside theheadphone earcup(s) (e.g., due to imperfect passive attenuation by theheadphones and earcup(s) when worn by the user, represented by waveform324, FIG. 3C) that is distinct from audio generated by and provided bythe headphones (e.g., distinct from device audio content, antiphaseaudio, and passthrough audio).

Taking into account the level of passive attenuation (e.g., provided byheadphones as physical barriers over a user's ears) when providingnoise-cancelling audio improves the noise-cancelling effect and reducesovercompensation for ambient sound, which in turn reduces overuse ofaudio circuitry. Providing an improved user experience and protectingaudio circuitry enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user toachieve an intended outcome and reducing user mistakes whenoperating/interacting with the device), and, additionally, reduces powerusage and improves battery life of the device by enabling the user touse the device more quickly and efficiently.

In some embodiments, while providing audio corresponding to audiocontent from the electronic device (818), audio corresponding to ambientsound from the respective physical environment is provided at anambient-sound audio level that is lower than an ambient-sound audiolevel at which audio corresponding to ambient sound from the respectivephysical environment is provided while not providing audio correspondingto audio content from the electronic device. For example, as describedherein with reference to FIG. 6A in comparison with FIG. 6E, whenproviding device audio content (e.g., shown by device content waveform606, FIG. 6A), the amount of ambient sound that is perceived (e.g.,shown by perceived ambient sound waveform 614, FIG. 6A) is less than theamount of ambient sound that is perceived (e.g., shown by perceivedambient sound waveform 614, FIG. 6E) when not providing device audiocontent (e.g., shown by device content waveform 606, FIG. 6E). One ofordinary skill will recognize that changing the ambient-sound audiolevel may be achieved by reducing the amount of noise-cancellationand/or increasing the amount of audio passthrough.

Providing less ambient sound when a user is listening to device audiocontent, compared to the amount of ambient sound provided when a user isnot listening to device content, balances the user's ability to hearprovided audio content against the user's ability to interact with thesurrounding physical environment, without requiring additional inputfrom the user. Providing an adaptive and more intuitive user experiencewhile reducing the number of inputs needed to achieve such an experienceenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by helping the user to achieve anintended outcome and reducing user mistakes when operating/interactingwith the device), which, additionally, reduces power usage and improvesbattery life of the device by enabling the user to use the device morequickly and efficiently.

In some embodiments, prior to detecting a second change in the one ormore audio properties of the respective physical environment (820), thewearable audio output device(s) detect a user input via the one or morewearable audio output devices (e.g., a gesture, such as a tap or swipe,performed on the one or more wearable audio output devices). Forexample, as described herein with reference to FIG. 6F, input 618 isdetected via headphones 604.

In some embodiments, in response to detecting the user input, the one ormore wearable audio output devices change one or more respective audiolevels of respective audio, for example to one or more respective presetaudio levels. For example, if a user input is detected whiledevice-content audio and ambient-sound audio are being provided atrespective levels determined (e.g., automatically, by the one or morewearable audio output devices) based on audio properties of therespective physical environment (e.g., as described herein withreference to FIG. 6F), the one or more wearable audio output devicestemporarily change the audio level of the device-content audio (e.g., asdescribed herein with reference to FIG. 6G) to a predefineddevice-content audio level (e.g., set by a user using an audio settingsuser interface on the electronic device), and the audio level of theambient-sound audio to a predefined ambient-sound audio level (e.g., setby a user using the audio settings user interface on the electronicdevice). In some embodiments, after a predefined time period sincedetecting the user input, the device reverts the respective audio levelsof the device-content audio and the ambient-sound audio from thepredefined (e.g., user-set) levels to levels determined (e.g.,automatically) based on audio properties of the respective physicalenvironment (e.g., as described herein with reference to FIG. 6J). Insome embodiments, regardless of whether the one or more wearable audiooutput devices initially change the respective audio levels of theprovided audio to predefined levels in response to detecting the input,generally, detecting the user input causes the one or more wearableaudio output devices to forgo changing (e.g., automatically) therespective audio levels of the provided audio during the predefined timeperiod since detecting the input (e.g., as described herein withreference to FIGS. 6G-6I).

In some embodiments, the wearable audio output device(s) detect thesecond change in the one or more audio properties of the respectivephysical environment. In some embodiments, in accordance with adetermination that the second change in the one or more audio propertiesof the respective physical environment is detected after a predefinedtime period since detecting the user input (e.g., a time period thatbegins when the input is detected), the wearable audio output device(s)change a respective audio level of respective audio (e.g., by changingthe device-content audio level of audio corresponding to audio contentfrom the electronic device, changing the ambient-sound audio level ofaudio corresponding to ambient sound from the respective physicalenvironment, and/or changing the audio-cancelling audio level of audioconfigured to cancel at least a portion of ambient sound from therespective physical environment). For example, as described herein withreference to FIG. 6J, detecting subsequent speech after the temporaryaudio output state has expired (and before another temporary audiooutput state is initiated) results in changes to respective audio levelsof respective audio provided by headphones 604 as described herein withreference to FIGS. 6A-6C.

In some embodiments, in accordance with a determination that the secondchange in the one or more audio properties of the respective physicalenvironment (e.g., person 622 speaking, FIG. 6H) is detected within thepredefined time period since detecting the user input (e.g., before timeT_(th), FIG. 6H), the wearable audio output device(s) forgo changing therespective audio level of the respective audio (e.g., as describedherein with reference to FIG. 6H). For example, in response to detectingthe input, the one or more wearable audio output devices are temporarilyplaced in a state in which automatic adjustment of audio levels ofrespective audio (e.g., the device-content audio level of audiocorresponding to audio content from the electronic device, theambient-sound audio level of audio corresponding to ambient sound fromthe respective physical environment, and/or the audio-cancelling audiolevel of audio configured to cancel at least a portion of ambient soundfrom the respective physical environment) is not performed in responseto detecting one or more changes in the one or more audio properties ofthe physical environment (e.g., as described herein with reference toFIGS. 6G-6I). In some embodiments, the wearable audio output device(s)detect multiple changes in the one or more audio properties of therespective physical environment, the multiple changes including a firstrespective change that is detected within the predefined time periodsince detecting a user input to initiate a temporary audio output state,and a second respective change that is detected outside of (e.g.,before, or after) the predefined time period since detecting a userinput to initiate a temporary audio output state.

Allowing a user to initiate a temporary audio output state, in which thelevels of different audio components in an audio output are notautomatically adjusted in response to changes in the audio properties ofthe surrounding physical environment, provides the user with additionalcontrol over audio output using a single input via the headphones ratherthan requiring the user to interact with a displayed user interface foradjusting audio settings. Reducing the number of inputs needed tocontrol audio output, and providing additional control options withoutrequiring a display to be powered on enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byhelping the user to achieve an intended outcome and reducing usermistakes when operating/interacting with the device), and, additionally,reduces power usage and improves battery life of the device by enablingthe user to use the device more quickly and efficiently.

It should be understood that the particular order in which theoperations in FIGS. 8A-8B have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,method 700) are also applicable in an analogous manner to method 800described above with respect to FIGS. 8A-8B. For example, the audiooutputs, simulated spatial locations, and physical environmentsdescribed above with reference to method 800 optionally have one or moreof the characteristics of the audio outputs, simulated spatiallocations, and physical environments described herein with reference toother methods described herein (e.g., method 700). For brevity, thesedetails are not repeated here.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to best use the invention and variousdescribed embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A non-transitory computer readable storage mediumstoring one or more programs, the one or more programs comprisinginstructions that, when executed by a wearable audio output device thatis in a respective physical environment and that is in communicationwith an electronic device, cause the wearable audio output device to:while one or more audio properties of the respective physicalenvironment satisfy first audio criteria, provide audio outputcorresponding to the first audio criteria, the audio output including:audio corresponding to audio content from the electronic device at afirst device-content audio level; and audio corresponding to ambientsound from the respective physical environment at a first ambient-soundaudio level; detect a change in the one or more audio properties of therespective physical environment; and in response to detecting the changein the one or more audio properties of the respective physicalenvironment, provide audio corresponding to ambient sound from therespective physical environment at a second ambient-sound audio levelthat is different from the first ambient-sound audio level.
 2. Thecomputer readable storage medium of claim 1, wherein the one or moreprograms include instructions that, when executed by the wearable audiooutput device, cause the wearable audio output device to: in response todetecting the change in the one or more audio properties of therespective physical environment, provide, via the wearable audio outputdevice, audio corresponding to audio content from the electronic deviceat a second device-content audio level that is different from the firstdevice-content audio level.
 3. The computer readable storage medium ofclaim 1, wherein: while the one or more audio properties of therespective physical environment satisfy the first audio criteria, theaudio output further includes audio configured to cancel at least aportion of ambient sound from the respective physical environment at afirst audio-cancelling audio level; and the one or more programs includeinstructions that, when executed by the wearable audio output device,cause the wearable audio output device to: in response to detecting thechange in the one or more audio properties of the respective physicalenvironment, provide audio configured to cancel at least a portion ofambient sound from the respective physical environment at a secondaudio-cancelling audio level that is different from the firstaudio-cancelling audio level.
 4. The computer readable storage medium ofclaim 3, wherein: detecting the change in the one or more audioproperties of the respective physical environment includes detecting anincrease in speech in the respective physical environment; and inresponse to detecting the change in the one or more audio properties ofthe respective physical environment, the second audio-cancelling audiolevel is zero.
 5. The computer readable storage medium of claim 1,wherein the audio output includes audio configured to cancel at least aportion of ambient sound from the respective physical environment, andthe audio configured to cancel at least a portion of ambient sound fromthe respective physical environment is based on an amount by whichambient sound from the respective physical environment is reduced by thewearable audio output device when worn by a user.
 6. The computerreadable storage medium of claim 1, wherein, while providing audiocorresponding to audio content from the electronic device, audiocorresponding to ambient sound from the respective physical environmentis provided at an ambient-sound audio level that is lower than anambient-sound audio level at which audio corresponding to ambient soundfrom the respective physical environment is provided while not providingaudio corresponding to audio content from the electronic device.
 7. Thecomputer readable storage medium of claim 1, wherein the one or moreprograms include instructions that, when executed by the wearable audiooutput device, cause the wearable audio output device to: prior todetecting a second change in the one or more audio properties of therespective physical environment, detect a user input via the wearableaudio output device; detect the second change in the one or more audioproperties of the respective physical environment; in accordance with adetermination that the second change in the one or more audio propertiesof the respective physical environment is detected after a predefinedtime period since detecting the user input, change a respective audiolevel of respective audio, the respective audio comprising the audiocorresponding to audio content from the electronic device, the audiocorresponding to ambient sound from the respective physical environment,and/or audio configured to cancel at least a portion of ambient soundfrom the respective physical environment; and in accordance with adetermination that the second change in the one or more audio propertiesof the respective physical environment is detected within the predefinedtime period since detecting the user input, forgo changing therespective audio level of the respective audio.
 8. The computer readablestorage medium of claim 1, wherein the wearable audio output deviceincludes one or more pose sensors for detecting a pose of the wearableaudio output device, and the one or more programs include instructionsthat, when executed by the wearable audio output device, cause thewearable audio output device to: detect a change in the pose of thewearable audio output device; wherein the audio corresponding to ambientsound from the respective physical environment is provided at the secondambient-sound audio level further in response to detecting the change inthe pose of the wearable audio output device.
 9. A wearable audio outputdevice that is in a respective physical environment and that is incommunication with an electronic device, comprising: one or moreprocessors; and memory storing one or more programs, wherein the one ormore programs are configured to be executed by the one or moreprocessors, the one or more programs including instructions for: whileone or more audio properties of the respective physical environmentsatisfy first audio criteria, providing audio output corresponding tothe first audio criteria, the audio output including: audiocorresponding to audio content from the electronic device at a firstdevice-content audio level; and audio corresponding to ambient soundfrom the respective physical environment at a first ambient-sound audiolevel; detecting a change in the one or more audio properties of therespective physical environment; and in response to detecting the changein the one or more audio properties of the respective physicalenvironment, providing audio corresponding to ambient sound from therespective physical environment at a second ambient-sound audio levelthat is different from the first ambient-sound audio level.
 10. Thewearable audio output device of claim 9, wherein the one or moreprograms include instructions for: in response to detecting the changein the one or more audio properties of the respective physicalenvironment, providing, via the wearable audio output device, audiocorresponding to audio content from the electronic device at a seconddevice-content audio level that is different from the firstdevice-content audio level.
 11. The wearable audio output device ofclaim 9, wherein: while the one or more audio properties of therespective physical environment satisfy the first audio criteria, theaudio output further includes audio configured to cancel at least aportion of ambient sound from the respective physical environment at afirst audio-cancelling audio level; and the one or more programs includeinstructions for: in response to detecting the change in the one or moreaudio properties of the respective physical environment, providing audioconfigured to cancel at least a portion of ambient sound from therespective physical environment at a second audio-cancelling audio levelthat is different from the first audio-cancelling audio level.
 12. Thewearable audio output device of claim 11, wherein: detecting the changein the one or more audio properties of the respective physicalenvironment includes detecting an increase in speech in the respectivephysical environment; and in response to detecting the change in the oneor more audio properties of the respective physical environment, thesecond audio-cancelling audio level is zero.
 13. The wearable audiooutput device of claim 9, wherein the audio output includes audioconfigured to cancel at least a portion of ambient sound from therespective physical environment, and the audio configured to cancel atleast a portion of ambient sound from the respective physicalenvironment is based on an amount by which ambient sound from therespective physical environment is reduced by the wearable audio outputdevice when worn by a user.
 14. The wearable audio output device ofclaim 9, wherein, while providing audio corresponding to audio contentfrom the electronic device, audio corresponding to ambient sound fromthe respective physical environment is provided at an ambient-soundaudio level that is lower than an ambient-sound audio level at whichaudio corresponding to ambient sound from the respective physicalenvironment is provided while not providing audio corresponding to audiocontent from the electronic device.
 15. The wearable audio output deviceof claim 9, wherein the one or more programs include instructions for:prior to detecting a second change in the one or more audio propertiesof the respective physical environment, detecting a user input via thewearable audio output device; detecting the second change in the one ormore audio properties of the respective physical environment; inaccordance with a determination that the second change in the one ormore audio properties of the respective physical environment is detectedafter a predefined time period since detecting the user input, changinga respective audio level of respective audio, the respective audiocomprising the audio corresponding to audio content from the electronicdevice, the audio corresponding to ambient sound from the respectivephysical environment, and/or audio configured to cancel at least aportion of ambient sound from the respective physical environment; andin accordance with a determination that the second change in the one ormore audio properties of the respective physical environment is detectedwithin the predefined time period since detecting the user input,forgoing changing the respective audio level of the respective audio.16. The wearable audio output device of claim 9, wherein the wearableaudio output device includes one or more pose sensors for detecting apose of the wearable audio output device, and the one or more programsinclude instructions for: detecting a change in the pose of the one ormore wearable audio output device; wherein the audio corresponding toambient sound from the respective physical environment is provided atthe second ambient-sound audio level further in response to detectingthe change in the pose of the wearable audio output device.
 17. Amethod, comprising: at one or more wearable audio output devices thatare in a respective physical environment and that are in communicationwith an electronic device: while one or more audio properties of therespective physical environment satisfy first audio criteria, providingaudio output corresponding to the first audio criteria, the audio outputincluding: audio corresponding to audio content from the electronicdevice at a first device-content audio level; and audio corresponding toambient sound from the respective physical environment at a firstambient-sound audio level; detecting a change in the one or more audioproperties of the respective physical environment; and in response todetecting the change in the one or more audio properties of therespective physical environment, providing audio corresponding toambient sound from the respective physical environment at a secondambient-sound audio level that is different from the first ambient-soundaudio level.
 18. The method of claim 17, including: in response todetecting the change in the one or more audio properties of therespective physical environment, providing, via the one or more wearableaudio output devices, audio corresponding to audio content from theelectronic device at a second device-content audio level that isdifferent from the first device-content audio level.
 19. The method ofclaim 17, wherein: while the one or more audio properties of therespective physical environment satisfy the first audio criteria, theaudio output further includes audio configured to cancel at least aportion of ambient sound from the respective physical environment at afirst audio-cancelling audio level; and the method includes: in responseto detecting the change in the one or more audio properties of therespective physical environment, providing audio configured to cancel atleast a portion of ambient sound from the respective physicalenvironment at a second audio-cancelling audio level that is differentfrom the first audio-cancelling audio level.
 20. The method of claim 19,wherein: detecting the change in the one or more audio properties of therespective physical environment includes detecting an increase in speechin the respective physical environment; and in response to detecting thechange in the one or more audio properties of the respective physicalenvironment, the second audio-cancelling audio level is zero.
 21. Themethod of claim 17, wherein the audio output includes audio configuredto cancel at least a portion of ambient sound from the respectivephysical environment, and the audio configured to cancel at least aportion of ambient sound from the respective physical environment isbased on an amount by which ambient sound from the respective physicalenvironment is reduced by the one or more wearable audio output deviceswhen worn by a user.
 22. The method of claim 17, wherein, whileproviding audio corresponding to audio content from the electronicdevice, audio corresponding to ambient sound from the respectivephysical environment is provided at an ambient-sound audio level that islower than an ambient-sound audio level at which audio corresponding toambient sound from the respective physical environment is provided whilenot providing audio corresponding to audio content from the electronicdevice.
 23. The method of claim 17, including: prior to detecting asecond change in the one or more audio properties of the respectivephysical environment, detecting a user input via the one or morewearable audio output devices; detecting the second change in the one ormore audio properties of the respective physical environment; inaccordance with a determination that the second change in the one ormore audio properties of the respective physical environment is detectedafter a predefined time period since detecting the user input, changinga respective audio level of respective audio, the respective audiocomprising the audio corresponding to audio content from the electronicdevice, the audio corresponding to ambient sound from the respectivephysical environment, and/or audio configured to cancel at least aportion of ambient sound from the respective physical environment; andin accordance with a determination that the second change in the one ormore audio properties of the respective physical environment is detectedwithin the predefined time period since detecting the user input,forgoing changing the respective audio level of the respective audio.24. The method of claim 17, wherein the one or more wearable audiooutput devices include one or more pose sensors for detecting a pose ofthe one or more wearable audio output devices, and the method includes:detecting a change in the pose of the one or more wearable audio outputdevices; wherein the audio corresponding to ambient sound from therespective physical environment is provided at the second ambient-soundaudio level further in response to detecting the change in the pose ofthe one or more wearable audio output devices.